Click to email me

Sitemapper
Data Logger

Have I got time for another project?  No, not really – but what the heck!  From time-to-time in my work I have a need to analyze public water supply pressures.  For this I need periodic measurements of pressure over several weeks.  The data is currently recorded using a circular chart recorder, but the thickness of the pen trace equates to 10’s of minutes on the polar axis and 10’s of kPa on the radial axis.  See Figure 1.  Further the electromechanical recorder’s damping (or should that be overshoot) make transients difficult to interpret.  Reading the circular charts is a pain at the best of times and interpreting the results is an art rather than science.

 

Figure 1.  Polar Pressure Plot

 

I have also been thinking about a high speed data logger for recording ultrasonic sound underwater to assist with the development of the Buddy Locator.   As it turned out the finished data logger has had more use in this application than in pressure recording.  You can read about the Buddy Locator and see some of field measurements obtained using this logger at
Aquacoustics.biz.

The pressure recorder is a typical data logger application.  The logger needs to be battery powered, accept up to eight input channels, incorporate a real time clock, provide at least 1% reading resolution, be capable of calibration, provide storage for at least two weeks of data taken at one minute intervals, and allow straight forward analysis using software such as MS Excel.

There are some very low cost basic SD card loggers available.   However the commercial units suitable for the pressure reading application are expensive and I have not been able to find something that is suitable for ultrasonic logging.  I decided to make one as a learning exercise and as an initial foray into high speed data acquisition.

A standard SD card formatted with FAT16 was selected as the data storage media because they are cheap, robust, small, and readily interface to a PC.

The next job was selecting a microprocessor.  I settled on an Atmel ATMega 163L which is regrettably now obsolete.  I chose this processor because I am familiar with the instruction set, I had one available (along with an Atmel SDK500 development board); it has sufficient SRAM for SD card sector reads, a 10 bit eight channel Analogue to Digital Converter (ADC), an SPI interface for an SD card interface and sufficient flash memory to allow development of a FAT16 file system.  The chip can be readily replaced, with minimal software changes, to more recent versions of the ATMega range such as the ATMega 16.

The hardware was designed with an LCD display, Real time Clock (RTC), four push buttons, 8 channel ADC, an LED and a serial port interface SD card interface through the SPI bus.

The board was developed using Eagle PCB software and included on-board regulated power supply, polarity protection, and filtering.  The board was made with an upper ground-plane to reduce noise on the ADC channels and eliminate jumpers.  See Figure 2.

The board was constructed and populated.  Software was developed in stages to test the hardware: first the LED, then the push buttons, then the LCD, followed by the RTC, then the serial port followed by the SD card interface, and finally the FAT16 file system.  While testing the hardware I wrote the primary subroutines in assembly language using AVR Studio.

 

Figure 2.  Data Logger

 

For development of the SPI to SD Card interface I give full credit to ‘the captain’.  The FAT16 routines were developed using information from Advanced Assembly Language by S. Holzner and P. Norton and Undocumented DOS  by Schulman, Michaels, Kyle Paterson, Maxy and Brown.  My initial SD card hardware test software allowed me to dump the SD Card sector-by-sector to my PC.  There was a time when operating systems allowed utilities such as Debug to access storage structures directly.  Alas Windows XP doesn’t allow this any more.

With all of the hardware working and tested I’ve started on the FAT16 file interface.  A significant problem is the capacity of MS Excel – 2^16 (65,536) rows of data * 2^8 (256) columns.  With one minute interval readings in each row this gives a maximum of 45 days of data.  It is intended to save the file as an ASCII text file using tab separation between readings.  Each set of readings is to be preceded by date and time rows and terminated with carriage return and line feed characters.  The maximum file size is therefore ~ 4 Megabytes corresponding with 7,680 * 512 byte sectors, or 480 * 16 sector clusters.

The software is developing somewhat slower than expected.  While the hardware interface routines were relatively easy to implement, the FAT interface has been somewhat more challenging due to the large number of 16 and 32 bit variables that are required to keep track of the FAT.  As at today the code is over 2000 instructions and I figure I have another 1000 to go.  My unfamiliarity with the development environment, AVR Studio, and C have not helped.  However I am making steady progress each evening and it will not be too long before I have completed the project in assembly language.

It’s 21 October already - where is the year going?  Life, the development of my Buddy Locator, my A20 camera housing, a slave underwater flash, and messing around with a robot have taken precedence over this project for too long.  I sat down today and started rethinking the file system design.  I am faced with three competing requirements, data security, write speed, and MS Excel compatibility.  At this time I have a hardware platform that cannot be readily modified to allow faster data transfer.  (If I’d thought about this more at the outset the design would have included a separate ADC, a paged RAM cache with its own MCU, and a storage device using the full width SD Card bus.)

The current design approach has been based upon a full FAT16 implementation with file updates on each 512 byte block store.  The limited RAM resource and the speed of the MCU are limiting factors.

Another option open to me is to ignore the FAT format.  This way I can write 512 byte blocks of data directly to the SD card,  read it back, block by block, convert it to an appropriate form, and send it via the serial port to a PC.  This would certainly be the fastest option for the current hardware and ensure data security for all but the last 512 byte block.  It is also the easiest solution to implement in software.

I have had another thought.  Why not initialize a full 4 MB Excel file using the FAT16 format?  Then I can write ASCII converted data full-tilt to the SD card without the file management read/write overheads.  If the file consists of sequential sectors then the write process should be very straight forward.  This will preserve the data security to the last 512 byte block while maintaining MS Excel compatibility, with optimized write speed for the current hardware.

24 October and I have started porting my assembler routines to C using WIN AVR.  At best I am a C novice and this experience has been painful.  Missing “}” and “_, and a lack of understanding of the dreaded Makefile have resulted in days of compiler errors and increased hair loss (let’s face it, I had little enough to start with).  I have been using a collection of references:  “C Programing for Microcontrollers” by Joe Pardue; “Embedded C Programming and the Atmel AVR”, by Barnett, Cox and O’Cull; and several books with the generic title C/C++ in ?? days.  Alas the literature has not kept pace with the Integrated Development Environment so changes in the header files have been frequent causes of grief.   That said I have used as many applicable code segments as I can find from the literature and the net in the hope of producing something that works this year.

I have managed to get the basic initialization and hardware functions implemented, compiled error free, and simulated in WIN AVR but I haven’t transferred these to the hardware platform yet.  The code has blown out from 2,000 odd bytes in assembler to over 5,000 bytes in C.  I’m rather hoping that the ‘include’ library overhead will not increase appreciably from here.

2 December 08 and another project has dictated a need for a fast data logger to assist with analysis of noise.  I have set about reprogramming the hardware to implement a 20 kHz 10 bit data logger.  The code has been implemented in assembler, but to achieve the desired sampling rate I have abandoned a FAT file format, reverting to an RS232 serial port data download via an assembler passer to convert the raw binary data into ASCII.  While this is a heap slower than a direct card data transfer it works.  Figure 3 shows a Microsoft Excel graph from a recorded 1 kHz square wave oscillating  between about 0.4V and 2.6V.  Figure 4 shows the original waveform.   With a standard 512 MB SD card I can record over 3 hours of data.

 

Figure 3.  Data Logger Output from 1 kHz Square Wave

 

Figure 4.  Square Wave Input
(Vert. 1 V/div, Horz. 0.5 ms/div)

 

I now have a tool that I can take to sea for recording the output of the Buddy Locator front-end electronics.

28 December and I’ve spent the last three days reprogramming the data logger to maintain the fast analogue to digital conversions but with a Windows compatible FAT16 file format on the SD card (reading a 10 Meg file at 19,200 bps was just too tedious) .  This has not been without its frustrations including my confusion between the  the LDS and LPM assembler instructions, little-endian and big-endian file formats, FAT16’s requirement for uppercase 8.3 file names etc, etc.  The code has been written entirely in assembler.  It isn’t as pretty as I’d like, but it works.  The file format is unsigned 2 byte binary (little-endian) for compatibility with Mat Lab for data analysis.  Click here to see the results - even if you aren’t, I was impressed!

I’ve had a few inquires about communicating with an SD Card and I’ve been happy to share my code with those that are interested, and for all for those that want it but are too shy to ask, here is.  Click on Code for the complete assembler file for the current fast logger application including Menu, SD card routines, FAT16 structure, and a debug sector dump utility.  Click on Schematic for the circuit diagram for the logger.  There is enough information here for you to make a fast, low-cost, SD card data logger that works without too much effort.

This information is provided free of charge, but without any warranty, guarantee, or statement of fitness for purpose.  All I ask is that you please acknowledge your source.

CODE                                 SCHEMATIC

                     

Although I have further plans for the data logger, this brings this particular project to a close.  Well, not quite.  I needed to put the data logger into a case before taking it into the big wide world.  As usual I couldn’t find an appropriately sized box so I spent a week making a plastic case out of 6 mm thick white Perspex.  Although the case isn’t water resistant, provided the SD card slot is sealed up with plastic tape, it should handle a few splashes.  See Figure 5.  Of course there were the unanticipated problems with what should have been a straightforward job.  The RS232 port side of the case took three attempts before I was satisfied with the work, I had to make two back plates when one of them cracked during drilling, and when I finally got around to assembling the old bits in the new case the logger refused to work.  I dreaded that I’d killed the microcontroller but it turns out I had damaged a track on the LCD card while removing it from the old PCB.  Nothing that an 0.8 mm copper rivet or two couldn’t fix.  All’s well that ends well!

 

Figure 5.  Ready for Action

 

2 February 2010 I have some plans for future development in line with my original requirements to enhance the ADC conversion rate for ultra-fast conversion.  I have 7 unused ADC channels on the existing board that are begging to be connected to real data sources.  Sometime soon I intend to utilize all of these in a low conversion rate (say 1 sample from each channel per second) universal data recorder.  The Atmel data-sheet fine print indicates that I can push the sampling rate up to about 77 kHz without appreciable conversion errors.

5 March 2012 and it’s been a while since I have had cause to do anything further to my logger.  A number of applications have come up in recent times requiring a USB 2.0 host for a flash card memory interface in FAT32.  There are a few nice to haves as well including a wireless interface and a DAC capability.  I’ve started on the USB host.  There are a number of approaches but a bit level interface (bit banging) won’t work fast enough and consumes too much processor time.  Even a serial interface between the processor and a USB engine is likely to be restrictively slow at 8 Mb/s given that low speed USB operates at 1.5 Mb/s.

3 September 2012 and the new USB based board has been designed and I have made a good start of the software.  The first module is a serial port boot loader because getting the USB embedded host working is going to take some development.

I’ll be reporting on progress from time to time but probably just letting you know what I have (and haven’t) achieved.  If you are interested in these developments then you will need to contact me.

[Turtles are Here] [Site Map] [750Z Housing] [Construction] [A20 Housing] [Drawings] [Galapagos] [Data Logger] [Making PCBs] [PCB Drill] [CD Welder] [Current Monitor] [Proportional Controller] [Lock In Amplifier] [NAD C 520] [Kenwood Tape Deck] [Tape to Computer] [Reflow Oven] [Designed to Fail] [M2 3D Printer] [Coil Winders] [Changing a Light Bulb] [Squash Racquet Repair] [Dive Photos] [Sand Casting] [Oxy-MAPP] [Lathe Maintenance] [Thermal Adhesive] [M73 Compass Repair] [Calcium Block for Turtles] [Infinite R Network] [ST7820 LCD] [Links] [Contact Me]