Which is Which? - Answers

Here are the answers to the question from the last post:

• Clip 1 is the original sound file
• Clip 2 is encoded with AES-ECB and scrambled lower 1 bits
• Clip 3 is has nothing encoded, but scrambled lower 1 bits

If you look at the clips visually with a sound editing program they all appear the same even at extreme zooms:


The audio clips sound the same as well, so how can you tell them apart? As it turns out the original WAV file compresses better than the files with the scrambled lower bits (43% vs. 38%). After you narrow down the data to clips 2 and 3 you could then extract the lower 1 bits from each and perform an analysis on them. The AES encyption method that we used (and are replacing) is block based and leaves a detectable pattern. Once you figure out how big the block of data is you could then turn to traditional means of breaking AES.

From here on we are working on replacing AES ECB with a better AES algorithm in an effort to reduce detectability. We are also investigating ways to reduce the randomness of our random data padding so that the compression ratio is the same as or similar to the original file's ratio.

Which is which?

Last night I took an audio clip and I ran it through Awesome-WAV two times. Once was with a 80 byte data file, and once was without any data (to scramble the lower bits).

The clips

Is it possible to determine which clip is which? If you can figure it out, then send an email to awesomewav@gmail.com with your answer. In your email note which clip is the original, which contains data, and which contains no data. Also let us know how you figured it out.

*It is pretty easy to determine which file is the original

awesome-wav 1.1.0

I am pleased to announce the release of awesome-wav 1.1.0. This is an intermediary release that mostly adds new features and fixes a few small bugs in behavior.

There are two major changes in this release. The first is supporting compression with quicklz as well as zlib, with options to not include zlib compression. This decreases compression time, though increases file-size by a bit. The other significant change is allowing wav files with unknown wav chunks. The current implementation just discards the unknown chunks, something that will be changed by the next major release.

This release also fixes various behavior bugs and will allow the chunks of the riff file to be read in any order instead of a fixed order.

awesome-wav 1.0.1

I am pleased to announce the release of awesome-wav 1.0.1. This is primarily a bug-fix release, fixing issues with wav file validation and a few other random minor bugs. This release also includes the somehow forgotten --version argument.

the argument processor was also back-ported from the under-development 1.1/1.2/1.5 code as it is a little more flexible and includes some other various code modifications that have no effect on the actual operation of the program.

cmake

I've been working on adding cmake as an option to assist with building awesome-wav. Currently the only target that works right (only been tested by me on Debian Squeeze) seems to work right at the moment.

If anyone would like to test it on mac os or windows, that would be greatly appreciated, and input on improving the CMakeLists.txt file would be appreciated as well.

How AES Encyption Works

I was doing some research a couple of days ago and I found a very nice visualization of the Rijndael Cipher. It was chosen as the Advanced Encryption Standard (AES) by the National Institute of Standards and Technology (NIST) in 2001.

awesome-wav 1.0.0

awesome-wav 1.0.0 has been released. It supports encoding into 8, 16, 24, and 32 bit PCM WAV files, 32 and 64 bit IEEE float WAV files, compressing the input data using zlib and data encryption with ECB AES.

Release includes 32 & 64-bit *nix binaries and 32-bit Windows binary.

awesome-wav download

Release 1.0 Prep

We've been fighting with a couple barriers to the planned 1.0 release of awesome-wav. Currently the biggest project is setting up cmake as suggested by David Doria among other things. Hopefully the release will be ready in the next week or so.

Recent Progress

Much progress has been made towards our goal of releasing version 1.0. This week we will be cleaning up our code and making last minute checks for memory issues, stupid code issues, and possible places where our program could break. The plan for now is to release 1.0 this Saturday or Sunday, after which we will begin making more major changes.

Back from Summer

And we're back! After a long summer of very slow development we are back to developing more and are currently cleaning up the code/comments/output to ensure usability. no new features will be added from here until we 'release' version 1.0 which should be in the next week or so after we do a cleanup of code.

After that, we will begin work on the next version which will have considerably better encryption, a compression library that can be built into the executable, actual large file support, and many other nifty features.

New cool things

There was an apparent lapse in our commits, but that was because I was working hard on a major code addition. Moments ago I made the last change that marks the addition of support for 256 bit AES encryption of data before being encoded into the wav file. I've been working on testing it and so far it has not been failing. I have also tested for memory leaks and we are happily free of them for the time being.

RCOS Update

Yesterday Tim and I gave a presentation at RCOS outlining our past and future progress on Awesome-Wav. Some new important features include: data compression, IEEE Float support, and buffering. In the future we hope to add data encryption and support for additional music formats. The slides can be found here.

Recent Updates

Since our last post we have fixed a lot of bugs in the newer parts of the code and finally totally resolved our encoding using 12 bits problems. We have implemented selecting a compression level, begun adding parts to support more kinds of wav files, and changed how we encode using four bits to a considerably faster method. Soon this method will be applied to encoding using 12 bits to make its time more comparable to the other methods.

Yet more progress

So today I re-wrote the code used to deal with options so that it looks cleaner and is easier to extend. After doing that I finished support for compressing and decompressing files before/after encoding using zlib. Future plans for compressing include being able to specify what level of compression you want to use.

Fixed many bugs

- Fixed the code for encoding / decoding at 12 bits per sample so it should actually be working properly now.

- Rewrote the code for figuring out the number of bits that we should encode / decode per sample. This should fix any problems associated with getMinBitsEncodedPS().

- Changed how we calculated to data buffer size to avoid certain issues.

Progress

The code rewrite is now well underway, thanks to tom, and the new system is much cleaner and uses better formulas. There is still a bug in the system for dealing with 8 bit wav files.

I have also started putting together compression support using zlib version 1.2.4. The code is in the repository and compiles on its own, but has not been tested or incorporated into the program as of yet.

bug discovered.

In the process of adding better 24-bit wav support and 32-bit wav support, I ran across a fairly large bug in our code that determines if the specified data file can fit in the wav file and how many bits should be used in the process. Tom and I talked and we have decided to change plans to fix this bug the right way by rewriting that part of the code. Until we finish fixing it, we won't be making many changes to improve wav file support.

Bug Fixes

We tested buffering a little more, but the most recent commits have all been fixing a comparability issue between 32 and 64 bit machines. There has also been a slight change of plan. We have decided that it would be in our best interest to write a program that tests the basic functionality of our program without us having to manually enter all the files and check them ourselves. This will be a more back-burner project, but it's development will most likely start shortly.

Buffering Complete

File buffering has been implemented in Awesome-WAV. The program will no longer use huge chunks of memory when manipulating WAV files and will no longer make excessive I/O calls when encoding / decoding data files.

We will continue making changes and optimizations to how we handle buffering until we are satisfied.

We have not yet tested all file combinations so the program may fail in some cases.

RCOS Introduction

Last Friday Tim and I gave our first RCOS presentation. We gave a brief introduction about Awesome-WAV; the slides can be found here.

Very busy

Tim and I have been very busy these past couple of weeks and as a result we haven't made as much headway as we would have liked. Luckily next week is spring break so expect more progress soon.

More recent update

Well it appears that the code is now reachable from the RCOS dashboard.

Progress is steady on adding buffering to the project. We came to an agreement about how to implement it, and now there are only a few changes left before it is implemented for encoding.

Buffering...

At the moment Awesome-WAV loads entire .wav files into memory before messing with the data.  Since PCM WAV files can easily reach 50MB+ in size we have decided that we must implement some sort of file buffering for encoding / decoding data.  We could just fread each individual byte, but that would add considerable overhead.  Instead we will read / modify / save several large chunks of data at a time (1MB blocks maybe?).  Right now we are just making structural changes to the code to make implementing buffering easier.

The google code SVN is finally working, though the RCOS dashboard is not cooperating with me. SO for anyone that wishes to actually view the source, go to http://code.google.com/p/awesome-wav/ instead.