DIY *nix computer w lots of ram and network: possible?

[usman]

I'd like opinions on the feasibility of a home project
successfully building a computer node that has ample
memory (1-4G), gigabit ethernet, the ability to run
some stripped down open unix-like kernel (tcp/ip and
nfs are sort of what I'm after) and nothing else. Specifically:

- Is this the kind of project that could be attempted
at home without formal training and a modest (1000s,
not 10s of 1000s) budget for tools, materials, etc?

- What processor could do the job? Would it be possible
to use off-the-shelf SDRAM or DDR DIMMS for memory?
Extremely cursory review suggest things like MC68030 -
is that way off base?

- How long do you think this project would take an
experienced professional? A complete newbie?

- Are there reference designs for this kind of system?

- The idea is to have a cluster of small, hopefully
inexpensive nodes that can be used in a diskless,
distributed filesystem. I see plenty of embedded
computers that could be used for this sort of thing,
but they typically have low memory and lots of
extra goodies: I want the opposite. Is DIY a good
alternative?

All comments appreciated!

 

[Master Raven]

I'd like opinions on the feasibility of a home project
successfully building a computer node that has ample
memory (1-4G), gigabit ethernet, the ability to run
some stripped down open unix-like kernel (tcp/ip and
nfs are sort of what I'm after) and nothing else. Specifically:

- Is this the kind of project that could be attempted
at home without formal training and a modest (1000s,
not 10s of 1000s) budget for tools, materials, etc?

- What processor could do the job? Would it be possible
to use off-the-shelf SDRAM or DDR DIMMS for memory?
Extremely cursory review suggest things like MC68030 -
is that way off base?

- How long do you think this project would take an
experienced professional? A complete newbie?

- Are there reference designs for this kind of system?

- The idea is to have a cluster of small, hopefully
inexpensive nodes that can be used in a diskless,
distributed filesystem. I see plenty of embedded
computers that could be used for this sort of thing,
but they typically have low memory and lots of
extra goodies: I want the opposite. Is DIY a good
alternative?

All comments appreciated!

Well I would go with off the shelf solutions myself. Look into the Micro
ATX format of motherboard. there are several that offer a host of
processor choices and some have on board chips like the Geo line, etc.
or look at the SBC (single Board Computer) format also available in the
ATX format and sized to fit in a 3.5 in drive bay. The advantage to the
SBC line is the fact they have lan, sound and video and more on board
not to mention the support for the PC 104 modules. these things are
small fast and can fit anywhere. Most support some kind of Disk on a
chip technology so you can have your OS at board level. check out some
of these links.

http://www.winsystems.com/
http://www.embeddedsys.com/
http://www.linuxdevices.com/articles/AT2614444132.html
http://www.applieddata.net/defaultSBC.asp

and not to mention our fav place EBay
http://search.ebay.com/single+board+computer

Let me know how you make out, I've been interested in doing something
similar.

R Morton

 

[Mac]

I'd like opinions on the feasibility of a home project
successfully building a computer node that has ample
memory (1-4G), gigabit ethernet, the ability to run
some stripped down open unix-like kernel (tcp/ip and
nfs are sort of what I'm after) and nothing else. Specifically:

- Is this the kind of project that could be attempted
at home without formal training and a modest (1000s,
not 10s of 1000s) budget for tools, materials, etc?

- What processor could do the job? Would it be possible
to use off-the-shelf SDRAM or DDR DIMMS for memory?
Extremely cursory review suggest things like MC68030 -
is that way off base?

- How long do you think this project would take an
experienced professional? A complete newbie?

- Are there reference designs for this kind of system?

- The idea is to have a cluster of small, hopefully
inexpensive nodes that can be used in a diskless,
distributed filesystem. I see plenty of embedded
computers that could be used for this sort of thing,
but they typically have low memory and lots of
extra goodies: I want the opposite. Is DIY a good
alternative?

All comments appreciated!

I don't think it is feasible for a newbie. Too many little things that can
go wrong.

For a seasoned professional, it would be feasible, but I doubt most
seasoned professionals would bother with it. You can buy something close
enough off the shelf.

Why don't you work on something else. How about a DSP board which
accepts analog stereo audio inputs and spits out mp3 data in UDP
packets.

Just a thought.

--Mac

 

[Don Taylor]

I'd like opinions on the feasibility of a home project
successfully building a computer node that has ample
memory (1-4G), gigabit ethernet, the ability to run
some stripped down open unix-like kernel (tcp/ip and
nfs are sort of what I'm after) and nothing else. Specifically:

- Is this the kind of project that could be attempted
at home without formal training and a modest (1000s,
not 10s of 1000s) budget for tools, materials, etc?

Google for Beowolf Cluster and see what you find.
By the time they are publishing books on how to do
things like this you can assume you have a chance.
Read everything you can before you start.

I was buying the under-$100 Fry's motherboard/2+ ghz AMD cpu/fan,
dropping in a pair of 512 meg ddram for under $100,
$2 used pci video cards and cdroms, $6 scrap atx cases,
$50 new atx power supplies. Get one cheap multiport
keyboard/video/mouse switch box and one cheap network hub.
Watch your temperatures, blow a lot of air through them.

They all boot off the cd's and run in the gig of memory.
Or you can boot over the network off a single drive.

Find yourself a local linux user group, try to find a
helpful person who isn't to rabid and who will answer
a few hundred of your questions. Choose a linux that
is known for being easy to setup.

 

[richard mullens]

Well I would go with off the shelf solutions myself. Look into the Micro
ATX format of motherboard. there are several that offer a host of
processor choices and some have on board chips like the Geo line, etc.
or look at the SBC (single Board Computer) format also available in the
ATX format and sized to fit in a 3.5 in drive bay. The advantage to the
SBC line is the fact they have lan, sound and video and more on board
not to mention the support for the PC 104 modules. these things are
small fast and can fit anywhere. Most support some kind of Disk on a
chip technology so you can have your OS at board level. check out some
of these links.

http://www.winsystems.com/
http://www.embeddedsys.com/
http://www.linuxdevices.com/articles/AT2614444132.html
http://www.applieddata.net/defaultSBC.asp

and not to mention our fav place EBay
http://search.ebay.com/single+board+computer

Let me know how you make out, I've been interested in doing something
similar.

R Morton

Sound advice I'd say.

I'm soon going to put together a new router that will connect my two networks together.

It will be an AMD64 on a ASUS A8V deluxe (which has a gigabit network card built in) + another gigabit card.

Pretty over the top it will have 1 Gig Ram and say 200 Gig Harddrive.

I'll use Fedora core 4 or Suse professional 9.3 for the 64 bit operating system.

I'll host my webserver there too which will be reachable from either network - one on cable and the other on ADSL.

I'll have some gigabit switches to connect to other machines.

I think this is easily doable. Just put it together and load up the software almost. What you want sounds ok too.
There are some micro ATX boards with 2 ethernet connections. Don't know if they are Gigabit though. This is better than my
idea which doesn't scale well.

 

[Rich Grise]

I'd like opinions on the feasibility of a home project successfully
building a computer node that has ample memory (1-4G), gigabit ethernet,
the ability to run some stripped down open unix-like kernel (tcp/ip and
nfs are sort of what I'm after) and nothing else. Specifically:

- Is this the kind of project that could be attempted
at home without formal training and a modest (1000s, not 10s of 1000s)
budget for tools, materials, etc?

- What processor could do the job? Would it be possible
to use off-the-shelf SDRAM or DDR DIMMS for memory? Extremely cursory
review suggest things like MC68030 - is that way off base?

- How long do you think this project would take an
experienced professional? A complete newbie?

- Are there reference designs for this kind of system?

- The idea is to have a cluster of small, hopefully
inexpensive nodes that can be used in a diskless, distributed
filesystem. I see plenty of embedded computers that could be used for
this sort of thing, but they typically have low memory and lots of extra
goodies: I want the opposite. Is DIY a good alternative?

All comments appreciated!

I'm confused here as to what you're looking for. First, it sounded
like you want a LAN or a router, but by the end, I'm wondering, "Web TV?
Playstations?? <scratches head>"

You can bet a bare-bones computer for next to nothing, with a MB
that will take up to 4GB RAM, I'm pretty sure, plug in a network card,
and load up just about anybody's distro of Linux - I guess there are a
couple that are designed to be stripped down, and I guess QNX or so is
optimized for embedded apps. But, for example, Slackware, which I use,
could be installed in just a couple of megabytes of disk, completely
stripped to the bones, although I'd want to put _some_ utility to it. ;-)
Like, terminals?

I guess what I'm getting at is, what are you trying to accomplish?

Thanks,
Rich

 

[Usman Muzaffar]

I guess what I'm getting at is, what are you trying to accomplish?

OK - what I'm really after is a large (say, >= 32G) ram disk: for the
kind of processing I do, I'm almost always IO bound, CPU idle. I want
to load my OS, programs and data into memory and stay there until I
sync to disk.

Vendors that make large solid-state drives are either slow or really
(>$10K) expensive. The closest commercial offering I've found to what
I want is this:

http://www.cenatek.com/product_rocketdrive.cfm

But it's only for Windows, takes proprietary DIMMS, and isn't going
over 4GB.

With 1GB DIMMs in the ~$70 range - I'm looking for a way to spend,
say, $3K to get 32G of filesystem space with microsecond, not
millisecond latency.

Ideally I'd like something like the RocketDrive, ie that plugs
directly into my PCI bus and appeared as a drive. That seemed both
very complicated and less scalable, so I was considering a cluster of
IP nodes - on a dedicated switch (basically, a not-totally-poor-man's
SAN).

Of course you're absolutely right - I could just string together cheap
PC mobos that each are maxed out at 4GB of ram -- and if I'm serious
about actually implementing this I probably will. But 8x(mobo + ps +
cpu) starts to get big, loud, and (possibly) more expensive than it
has to be. Those mobos will have so much extra stuff on them that's
not needed for the application, that I was wondering what the
feasibility was of constructing a node that has nothing but RAM and
disk, that could lead to a cleaner, smaller, and hopefully cheaper
solution.

Hopefully that's clearer - any thoughts?

thanks,
-usman

 

[Alan Turner]

Hi Usman,

But it's only for Windows, takes proprietary DIMMS, and isn't going
over 4GB.

With 1GB DIMMs in the ~$70 range - I'm looking for a way to spend,
say, $3K to get 32G of filesystem space with microsecond, not
millisecond latency.

You can have lots of fast memory on a PC - that much is easy enough.
Fast access to this memory will surely be the difficult part.

How fast can you move data over 1Gb ethernet when you factor in the
efficiency of the protocols above the link layer? I bet modern hard
disks would give this ramdisk scheme a run for it's money.

If you can offload some of the data processing onto the individual nodes
such that data is moved there, processed, and the results moved back -
then you may win.

Hope this helps.

Regards,
Alan

 

[Philip Frampton]

I'm after plans for building a >300khz ADV converter. The use it in
recording from an ultrasonic microphone to a portable computer.

The interest in being able to accurately record bat calls. I have
heterodyne and frequency division systems for real time listening, but
for accurate analysis an exact record of the sound is required.

Looking for a commercial solution leads to either a very expensive

10Mhz mains powered solution, or a cheap <10khz version.

What would be ideal is a project that could be somehow coupled to a
palmtop computer (eg Sharp Zaurus) or if not, via USB to a laptop.

Many thanks for any suggestions,
Philip

PS This is an amateur interest.

 

[Joel Kolstad]

With 1GB DIMMs in the ~$70 range - I'm looking for a way to spend,
say, $3K to get 32G of filesystem space with microsecond, not
millisecond latency.

Ideally I'd like something like the RocketDrive, ie that plugs
directly into my PCI bus and appeared as a drive. That seemed both
very complicated and less scalable, so I was considering a cluster of
IP nodes - on a dedicated switch (basically, a not-totally-poor-man's
SAN).

You'd be pretty unlikely to get microseond latency by the time you're going
through your own PC's network stack, the switch, and your storage device...

Still, I like your idea. It would be pretty simple to build an "Ethernet hard
drive" at the hardware level though... just some Ethernet controller IC and
possibly even just an FPGA could do the actually memory access -- no processor
needed. This approach forces you to put all the "intelligence" in a driver on
the PC side, though -- and writing a file system driver for Windows is a
HIGHLY non-trivial undertaking! A much easier solution would be to make your
peripheral appear to Windows as a SCSI drive... writing a SCSI driver is not
particularly difficult (Windows gives your driver the complete SCSI control
block and says, "make it so!" -- i.e., do whatever you need to do to get the
drive to "hear" the command), and then you just need to hook your SCSI device
driver up to a system Ethernet driver somewhere. What happens, then, is that
your RAM disk peripheral will literally end up being formatted as FAT32 or
NTFS or whatever. This approach also makes it easier to be able to use the
peripheral from other OSes (Linux, Mac OS, etc.)

The overall easiest solution would perhaps be to hack into some single board
Linux machine (such as those used in wireless access points) and only write a
Linux device driver (which is much simpler than a Windows device driver, by
the way) for a bunch of RAM you somehow 'grafted' onto the board. Now your
peripheral immediately works with Windows networking, NFS, etc.... cool!
....but the downside is that every I/O request has to go through the entire
Linux networking stack, so your latency is going to be much higher (probably
milliseconds...), although with some tweaking you'll still have a tremendous
throughput advantage over rotating media.

---Joel Kolstad

 

[Tim Shoppa]

The idea is to have a cluster of small, hopefully

inexpensive nodes that can be used in a diskless,
distributed filesystem.

From most DIY to least DIY, look at PC/104, mini-ITX, and Wyse

WINterms.

Also google the term "dickless workstation".

Tim.

 

[usman]

Still, I like your idea. It would be pretty simple to build an "Ethernet hard
drive" at the hardware level though... just some Ethernet controller IC and
possibly even just an FPGA could do the actually memory access -- no processor
needed. This approach forces you to put all the "intelligence" in a driver on
the PC side, though -- and writing a file system driver for Windows is a
HIGHLY non-trivial undertaking!

Right, this is the other approach - I'd much prefer this, really - my
background is software and fs drivers sound a lot easier to me than
hardware.

I had ruled this out because I didn't think I could build a node with
large RAM without making it a full blown system anyway - can you tell
me a bit more about what this might look like? Could I scale it to 4GB
/ disk? Could I use off-the-shelf DDR DIMMS?

Thanks for any comments/answers!
-usman

 

[Don Taylor]

OK - what I'm really after is a large (say, >= 32G) ram disk: for the
kind of processing I do, I'm almost always IO bound, CPU idle. I want
to load my OS, programs and data into memory and stay there until I
sync to disk.

Ah, the REAL question.

Vendors that make large solid-state drives are either slow or really
(>$10K) expensive. The closest commercial offering I've found to what
I want is this:

But it's only for Windows, takes proprietary DIMMS, and isn't going
over 4GB.

There are 64 bit motherboards that don't have the 4GB limitation,
they just cost an order of magnitude more than the usual boards,
and often don't accept the bargain basement cheap pc memory sticks.

With 1GB DIMMs in the ~$70 range - I'm looking for a way to spend,
say, $3K to get 32G of filesystem space with microsecond, not
millisecond latency.

You'd perhaps want to consider implementing all your OS drivers
in VHDL, and maybe the whole OS. The layer after layer of software
is likely to swamp the amount of time it takes.

Ideally I'd like something like the RocketDrive, ie that plugs
directly into my PCI bus and appeared as a drive. That seemed both
very complicated and less scalable, so I was considering a cluster of
IP nodes - on a dedicated switch (basically, a not-totally-poor-man's
SAN).

Ask what the real typical throughput is for their drive, not some
hypothetical burst rate.

Of course you're absolutely right - I could just string together cheap
PC mobos that each are maxed out at 4GB of ram -- and if I'm serious
about actually implementing this I probably will. But 8x(mobo + ps +
cpu) starts to get big, loud, and (possibly) more expensive than it
has to be. Those mobos will have so much extra stuff on them that's
not needed for the application, that I was wondering what the
feasibility was of constructing a node that has nothing but RAM and
disk, that could lead to a cleaner, smaller, and hopefully cheaper
solution.

It seems unlikly that you are going to get a working system that is
cheaper than the commodity PC boards with 4 gig each.

 

[keith]

Ah, the REAL question.


There are 64 bit motherboards that don't have the 4GB limitation,
they just cost an order of magnitude more than the usual boards,
and often don't accept the bargain basement cheap pc memory sticks.

This board (Tyan S2875S) takes 8GB (4 x 2GB sticks) and didn't cost
anywhere near an order of magnitude more than any other 64bit board. I
bought it a little over a year ago for about $200. It does take
registered DIMMs, but apples to apples they aren't all that much more
expensive at the smaller sizes ($70 vs $63 for 512MB, DDR400). I don't
see 2GB unbuffered DIMMs though:

http://www.crucial.com/store/listmodule.asp?family=DDR&tabid=DDR+PC3200#184-pin DIMM

You'd perhaps want to consider implementing all your OS drivers in VHDL,
and maybe the whole OS. The layer after layer of software is likely to
swamp the amount of time it takes.

Why nolt add more processors? A 4-way Opteron board should be able to
handle 32GB natively. ...as long as we're talking a money-pit. ;-)

Ask what the real typical throughput is for their drive, not some
hypothetical burst rate.

The burst rates on the outter tracks should be sustainable. Of course
if you have lotsa small files you have that software uglyness again. The
trick is staying on the outer tracks. ;-)

It seems unlikly that you are going to get a working system that is
cheaper than the commodity PC boards with 4 gig each.

Or not-so-commodity with 32GB each, for that matter.

 

[Don Taylor]

This board (Tyan S2875S) takes 8GB (4 x 2GB sticks) and didn't cost
anywhere near an order of magnitude more than any other 64bit board. I
bought it a little over a year ago for about $200. It does take
registered DIMMs, but apples to apples they aren't all that much more
expensive at the smaller sizes ($70 vs $63 for 512MB, DDR400). I don't
see 2GB unbuffered DIMMs though:

http://www.crucial.com/store/listmodule.asp?family=DDR&tabid=DDR+PC3200#184-pin DIMM

The original poster was asking for a board that would support 32 gig.
Those boards are out there but they aren't $200. Even fairly ordinary
Opteron boards were $400+. And he was quoting commodity prices for
fairly cheap ram but wanting a board that would run 32 gig.

Why nolt add more processors? A 4-way Opteron board should be able to
handle 32GB natively. ...as long as we're talking a money-pit. ;-)

Because he was asking for microsecond response to get blocks off the
"disk" AND he didn't want to throw unlimited money at it, he was asking
for sale price memory to be (roughly) the whole cost of the system and
still get blocks off the disk in a microsecond.

The burst rates on the outter tracks should be sustainable. Of course
if you have lotsa small files you have that software uglyness again. The
trick is staying on the outer tracks. ;-)

And he isn't going to get a sustained rate of a block a microsecond.
Half a dozen years ago someone was trying to get "real time" performance
out of available "finger speed" OS. They did some measurements and found
it not too surprising to have 100,000 microsecond variations in response.

 

[keith]

The original poster was asking for a board that would support 32 gig.
Those boards are out there but they aren't $200. Even fairly ordinary
Opteron boards were $400+.

"Farily ordinary Opteron boards" are $200, or less.

And he was quoting commodity prices for
fairly cheap ram but wanting a board that would run 32 gig.

I understand. I want a Maseratti for $50 too. Would you design and build
me one?

Because he was asking for microsecond response to get blocks off the
"disk" AND he didn't want to throw unlimited money at it, he was asking
for sale price memory to be (roughly) the whole cost of the system and
still get blocks off the disk in a microsecond.

Wants, wants, wants. I want the above Maseratti too.

And he isn't going to get a sustained rate of a block a microsecond.
Half a dozen years ago someone was trying to get "real time" performance
out of available "finger speed" OS. They did some measurements and
found it not too surprising to have 100,000 microsecond variations in
response.

Sure, That makes sense, since 100msec is on the order of the OS's
context switch time.

Yup

Dreamers are everywhere. ;-)

 

[Don Taylor]

"Farily ordinary Opteron boards" are $200, or less.

Ooo... where and which ones? I thought I checked Pricewatch and only
saw those that were 2x that or more. I wanted to try out Solaris 10
but that only works on Opterons and Sparc's. For me a slow 1-way
Opteron and a board that would accept commodity cheap ram would have
been enough, but I'm trying to not have high expectations.

I understand. I want a Maseratti for $50 too. Would you design and build
me one?

He can certainly buy himself a server board that would support 32 gig.
And he could put a 64 bit OS on it, open up his 32 gig file and put it
in one big block of memory and skip all file system operations after
that. That would give him microsecond access to the whole block.

Slower than that, but not abandoning the file system completely was
the Solaris "mapping a filesystem into memory." That's the equivalent
of "ram drive" in PC-speak. But that must pay for software overhead.

Wants, wants, wants. I want the above Maseratti too.

Well, that was him.

Sure, That makes sense, since 100msec is on the order of the OS's
context switch time.

I think some of the Windows timings were substantially worse than that,
not having pre-emptive switching at that point in time.

Dreamers are everywhere. ;-)

Everybody needs a hobby

 

[Wouter van Ooijen]

I'm after plans for building a >300khz ADV converter. The use it in

recording from an ultrasonic microphone to a portable computer.

The Philips ARM chips have 400 kHz A/D converters.



Wouter van Ooijen

-- ------------------------------------
http://www.voti.nl
Webshop for PICs and other electronics
http://www.voti.nl/hvu
Teacher electronics and informatics

 

[keith]

Ooo... where and which ones? I thought I checked Pricewatch and only
saw those that were 2x that or more. I wanted to try out Solaris 10
but that only works on Opterons and Sparc's. For me a slow 1-way
Opteron and a board that would accept commodity cheap ram would have
been enough, but I'm trying to not have high expectations.

I paid about $200 for this Tyan S2875S(single CPU) and the S2875 (dual
CPU) was about $30 more (I wish I'd gone for it). Looking at NewEgg, it
seems that socket-940 isn't all that popular anymore (they no longer
carry Tyan), though they have a FoxCOnn board for $224. NewEgg lists the
Tyan S2850 (also the AMD chipset) from for $218.

Why wouldn't Solaris run on an Athlon64? They're much cheaper and do take
cheap RAM.

He can certainly buy himself a server board that would support 32 gig.
And he could put a 64 bit OS on it, open up his 32 gig file and put it
in one big block of memory and skip all file system operations after
that. That would give him microsecond access to the whole block.

Slower than that, but not abandoning the file system completely was the
Solaris "mapping a filesystem into memory." That's the equivalent of
"ram drive" in PC-speak. But that must pay for software overhead.



Well, that was him.
Understood.




I think some of the Windows timings were substantially worse than that,
not having pre-emptive switching at that point in time.

Sure. "Cooperative multitasking" is oxymoronic.

Everybody needs a hobby

;-)

 

[[email protected]]

I'd like opinions on the feasibility of a home project
successfully building a computer node that has ample
memory (1-4G), gigabit ethernet, the ability to run
some stripped down open unix-like kernel (tcp/ip and
nfs are sort of what I'm after) and nothing else. Specifically:

- Is this the kind of project that could be attempted
at home without formal training and a modest (1000s,
not 10s of 1000s) budget for tools, materials, etc?

- What processor could do the job? Would it be possible
to use off-the-shelf SDRAM or DDR DIMMS for memory?
Extremely cursory review suggest things like MC68030 -
is that way off base?

- How long do you think this project would take an
experienced professional? A complete newbie?

- Are there reference designs for this kind of system?

- The idea is to have a cluster of small, hopefully
inexpensive nodes that can be used in a diskless,
distributed filesystem. I see plenty of embedded
computers that could be used for this sort of thing,
but they typically have low memory and lots of
extra goodies: I want the opposite. Is DIY a good
alternative?

All comments appreciated!

1) Buy a big plate. Mount 8x PSU, Motherboard with ram+cpu+network.
Join them with a gigabit switch. Boxes just obstruct cooling. And saves a
few bucks too
ABIT AV8 S939 can hold upto 4 GByte.

2) Program a FPGA to become a 32GiB memory controller (35 bit address line).
Use cheap of the shelf sdram/ddr etc..
Add a scsi U160, U320 interface or some Sata interface. Which avoids the
trouble writeing a device driver. Aswell as network stack latency..

Just a quick thought. Not checked for obstacles makeing the project(s)
unfeasable or unreasonable :=)