Linux: Reading the Instructions

Questions, answers, and news related to computer hardware and software

Linux: Reading the Instructions

Postby Yogi » 09 Apr 2013, 15:19

Have you ever read the instructions on how to use Linux? I'm not talking about some obscure command to which you don't know the attributes or variables. Have you ever read how to point and click, or what to do in the terminal window? Well I never read that basic stuff because I knew all about it before I started my adventures in the Linux environment - well, I though I knew all about it.

You people who have a lot of familiarity with Linux are going to smile at my story. Actually, I suspect it will be more of a smirk. But hey, I don't mind being laughed at. I laugh at myself all the time, which is why I'm writing about this discovery I made today. I discovered that copy and paste in a Linux terminal is not the same as it is in the legacy operating system. That's the euphemism the author of the tutorial I just read uses for MS Windows. Anyway, why would something as basic as copy and paste be different in Linux? The answer is to save mouse clicks and to get on with the real work.

The Linux copy and paste could be done as you would do it in legacy operating systems, but all you have to do is highlight the text you want to copy, move your pointer over to the terminal window and press the middle mouse key. Done. All in one click. I've been getting into the bowels of Linux lately, and I can't tell you how many unnecessary mouse clicks and selections in drop down menus I've executed in order to work that CLI (command line interface). I was astounded to see how well this highlight and click works. Now that I know how to do it, it will add a minimum of 2 seconds to the productive work I could be doing each day instead of clicking mouse buttons.

Feel free to add your snide remarks. I deserve them. :nana:
User avatar
Yogi
Oracle Class Poster
Oracle Class Poster
 
Posts: 7013
Joined: 04 Aug 2007, 19:37
Location: Chicagoland

Re: Linux: Reading the Instructions

Postby Kellemora » 10 Apr 2013, 08:09

You're doing better than me then Yogi!
I've yet to find any tutorial helpful.
They all assume you already have a PhD in computer science and no less than a masters degree in programming.

An actually, what you find beneficial, is a pain in the arse to me.
When I paste a line into Terminal, it automatically executes that line.
The problem here is, I was not ready to execute the line yet. I need to cut n paste something else to it first.
So, I have to open and use a text editor first, assemble the instructions there, then copy it, go and paste the whole command at once in Terminal.

So, where you are saving a mouse click, others of us have several extra steps added. ☺

TTUL
Gary
User avatar
Kellemora
Brainiac Class Poster
Brainiac Class Poster
 
Posts: 3389
Joined: 07 Jul 2012, 18:52
Location: From St. Louis, current Knoxville, TN

Re: Linux: Reading the Instructions

Postby Yogi » 10 Apr 2013, 11:22

Try this to see if it helps: http://www.linuxcommand.org/index.php

As far as executing upon paste goes, you are experiencing something unusual. The paste into a terminal window operation should not execute until you hit the "Enter" key, thus giving you the opportunity to edit the line if necessary. I've looked through my Ubuntu Xterm settings and found nothing obvious that could change that behavior. There are several versions of the terminal program available in most distributions of Linux. Try changing to one of the others and see if that helps. Xterm never gave me a problem which is why I like to use it.
User avatar
Yogi
Oracle Class Poster
Oracle Class Poster
 
Posts: 7013
Joined: 04 Aug 2007, 19:37
Location: Chicagoland

Re: Linux: Reading the Instructions

Postby Kellemora » 11 Apr 2013, 10:25

Thanks Yogi - I bookmarked that website!

I also checked something in Terminal.

If I am careful to copy a line of code, letter for letter, and stopping at the last letter, it does not automatically run.
But if I hit Shift-End to copy the line, then it does run.
For the most logical reason too! There is a carriage return at the end of each line I saved in a text file.
So, if I hit Shift-End, then hit the left arrow key once while holding down the shift key, then it don't run automatically.

Sometimes I feel SO DUMB, hi hi.....

Since we are getting ready to go on a week long trip, bobbing around in the ocean.
I thought I would copy my (writing) working files over to the notepad.
I do have both Win7 and Debian6 on the notepad computer.
But since I cannot get WIFI to work on Debian6, and it works fine on Win7, I did two things.
I took my spare empty partition (which is a logical drive) and formatted it to NTSF.
Win7 sees it and assigns it drive E. and I can access it no problems.

But dig this.
I copied this file onto the spare partition using Debian6, this will be my untouched master file.
I went into Debian6 and copied the entire folder to my Desktop.
When I went into Win7 to copy the entire folder to my Win7 Desktop. It could not do it.
It claimed, FILE NAME TOO LONG, SKIP or CANCEL.

OK, I went directly to the Original File folder on the NAS, from Windows7 using WiFi.
Got the same ERROR message.

I went through every single File and the LONGEST ONE is only 54 characters long.
All of the rest are under 30 characters in length.
Yet the one that is 54 characters long DID COPY to the Desktop.

I go on-line to see what's up with Win7 not able to read a filename that is only 27 characters long.
I learned that one heck of a lot of folks are having these types of problems with Win7.
People moving from XP to Win7 can't copy a lot of their files. Unless they use command line to do it.
Lots and lots of commentary about how lousy Win7 is, ONE a Windows Only website.

In any case, even though Windows claims you can have 32 nested folder.
It counts every letter and space of each folder in its 260 character file name limitation.

To prove this, I made a simple text file and named it "Z.txt" and placed it in one of the folders I get 17 "file too long" error messages from. As you can see, the file name is only a total of 5 characters long. But it is TOO LONG for Win7 to recognize.

Now, if I go directly to Drive E. and through the folders to the file named Z.txt or any other file in that folder. It opens with no problem. And I can copy the file directly to the desktop with no error message.
But if I try to place it into the desktop folder, or the drive E folder where it belongs. I get the error message again.

Now, if I open the file directly from drive E and change a word or two in it and hit the save button. I get a different error, "Unable to Save" and if I try again I get "Path Not Found" or something like that.
If I make the same change to the document on the desktop, not in any folder. It saves just fine.

Move back over to Debian6 and everything works as it should. No problems with copying or saving files.

So, I decided to check something else that is 100% Windows and always has been.
I installed my Family Tree Maker program on Win7.
Using a Windows XP desktop, I copied my current Genealogy file folders to the NAS, which is NTFS, using Windows Explorer.
I move to the notebook and open Win7, go to the NAS to copy the Genealogy File Folder to the Windows Partition on the notebook. Again using Windows Explorer. I had over 480 files that said "FILE NAME TOO LONG, SKIP or CANCEL."

I opened the Genealogy Program itself, and directed it to read from the NAS. It did with no problems. When I closed the program, it made its automatic backup of the file onto the NAS. No problems.

I deleted the partial file from the Windows Partition.
Set the Drive E partition to SHARE.
Went back to the XP Computer and copied the Genealogy File to the notebook computers Drive E.
Opened my genealogy program in Win7 on the notebook computer. Pointed it to Drive E and opened the genealogy file. No problem opening the file. I did not make any changes, just closed the program and got the warning message "Unable to Back Up File xxxxx.xxx Destination Path Not Found." I checked the destination path, and it was the same folder it just read the working file from.
I tried copying the folder from Drive E to the desktop and got the "File Name Too Long," error message on the same 480 files.

OK, let's try something else. Back to the Windows XP desktop, open the genealogy program. Tell it to open the file on the notebook shared folder in Drive E on that machine. It opened just fine. Made no changes, and when I closed the program, it ran it's automatic backup. So now, on drive E I have both the main file and the backup file. No problems.
Go back to the notebook. Open the Genealogy Program, open the file on Drive E. It appears no problem. I check a few files, move around to several areas of the program, everything is A-OK. I close the program and the backup FAILS giving the same error messages as before, destination path not found.
This time I tried something else. I changed the Destination Path to a New Folder I created on the desktop, simply named, Test Folder One. Opened and when I went to close the program, I changed it's destination path to the new folder on the desktop. It saved the backup with no problems to this folder. No file name too long error messages.
However, I opened the genealogy program FROM the backup file and let it rebuild itself. Then told the program to save to the original to Test Folder One. I get the error message "File Name Too Long" for saving the original. But as the program closes, it backs up the file with no problems.

There is something totally amiss with Win7!!!!!
And the web is loaded with complaints about this problem.
The only solution is to make a single folder one folder ahead of the root directory and shorten any long file names by hand.
Which blows the whole purpose of having nested file folders to organize your files.

What I've learned I can do, is to make a Folder on Drive E and call it something like, File Transfer Folder, and move whatever document I want to work on into that folder, then move it to the desktop. And when finished with my work, boot into Debian6 to put it back where it belongs.

Sorry the rant turned out rather long.

TTUL
Gary
User avatar
Kellemora
Brainiac Class Poster
Brainiac Class Poster
 
Posts: 3389
Joined: 07 Jul 2012, 18:52
Location: From St. Louis, current Knoxville, TN

Re: Linux: Reading the Instructions

Postby Yogi » 11 Apr 2013, 12:18

No need to apologize for the rant. I got lost early on due to the complexity of what you are tying to do, but I get the general idea because I have experienced it myself. :grin:

Well, not the exact problem, but something close. I have a Linux based NAS which is primarily used for backups. All the volumes on the single hard drive are Windows shares. All the Windows based computers here have no trouble reading and writing to that NAS. When I started experimenting with Linux, back in the Ubuntu 9.4 days, there was a problem with samba and winbind that had to be corrected manually. All I needed to do was add some information to a configuration file, but it took months to figure that out. At any rate, as Ubuntu matured the errors it has with Windows shares disappeared. I now have nearly no trouble reading and writing to the NAS from any instance of any operating system I've tried to use.

Then one day while in Ubuntu I tried to move a shortcut over from the Unity desktop to the NAS. Nothing happened. So I tried a second time and got a warning that the file already exists - do I want to replace it. It might have existed but I could not see it from Ubuntu. Then I opened an instance of Windows and looked at the directory on the NAS, and Windows could not see any such shortcut either. HMMMM. :think:

So, out of instinct, I decided to change the name of the shortcut on the Unity desktop. I gave it a single character name, something like X with the appropriate extension. Voila! It copied over with no objection. Plus Windows7 was able to see it. After that unexpected success, it dawned on me that something might be amiss in the name of the shortcut file. So I took the original and kept deleting characters from it's file name until it showed up on the NAS. I'm guessing there is a long name/short name file issue involved and that Linux is adding invisible characters (such as spaces) to it's file names. The conventions for file names in Windows is not the same as those in Ubuntu (maybe all of Linux), but I have yet to discover what conventions are being violated. I do get that "too long file name" error once in a while, but only from the Linux side. All I have to do is rename the file to something shorter to fix the error.

You bring up the issue of full path file names. I suppose there is some kind of limit there, but I rarely run into it. When I do backups and system security scans the directory paths get to be quite lengthy. I have noticed that I can copy anything over to the NAS from Windows, but copying it back (or deleting the files) may require going down to the base level of the directory path in order to perform the task. While it's typically a Windows directory structure in which I've seen this happen, I have run into the exact same problem on the Linux server that hosts this web site. Some deeply nested files cannot be deleted unless I go some depth into the nest to delete them.

Life would be a lot easier for us geeks if there were such a thing as standards, eh?
User avatar
Yogi
Oracle Class Poster
Oracle Class Poster
 
Posts: 7013
Joined: 04 Aug 2007, 19:37
Location: Chicagoland

Re: Linux: Reading the Instructions

Postby Kellemora » 12 Apr 2013, 10:26

Back in the days of DOS, I think we were limited to file names of only 7 characters? Maybe it was more, I can't remember that far back. But I do remember maintaining a text file for all of the automatically shortened file names that used a Tilde1, Tilde2, etc. at the ends of the file names.

I had another problem in Win95, where we could use longer file names. msWORD allowed you to save files, that were invisible to Windows Explorer or Notepad. Window WRITE could see them.
Thankfully, there was a failsafe that would not let you delete a folder that was not empty.
Because of that feature, I found hundreds of files I thought were lost forever, some of them were important files too.

Eons ago, when I first tried RedHat Linux, I grabbed what I thought was an empty hard drive off the shelf.
At least I had taped a note to it, that I moved all the files to the new Win98 computer. With a backup on another drive I had sitting there.
For some reason I decided to compare the two drives before reformatting one. I got a different report from each one.
So, I began checking and comparing the files. Again, I found about 30 files that didn't transfer to the backup or over to the Win98 computer. And was able to rescue those by going all the way back into DOS and renaming the files.

I may be wrong here, but it seems like, in Windows, you could not start a file name using a number. So I got in the habit of using the letters ymd for year, month, day, ahead of each file I wanted to keep in date order.
I still do this only now the date comes first, then the ymd as a memory jogger.
Interestingly enough, all of the publishers I have worked for over the years, maintain files in the same way.

The part that kills me the most, is I can rename a file that was claimed to have too long a filename to a single letter, and it still will not move that file, claiming it is too long.

And the most baffling thing of all. Yesterday while I was typing the previous message, I opened a file from the desktop, and it opened. Today it is telling me, file name too long. What happened between yesterday and today?

We are getting ready to go on a cruise, leaving on the 21st. I'm bringing the notepad with me, so I can do some work if I get bored. Since I can't get WIFI working on the Debian side, and it works just fine on the Windows side, I thought I would put my working folder in each OS, just in case I wanted to upload a file through WiFi to an editor, probably through Dropbox.

I'm thinking of buying a USB dongle, but none of the sellers disclose what chipset is in them, for me to compare it with the Linux drivers listings.

I think I mentioned I used an empty partition and set it ntfs and copied my folders to it, and Win7 still said too long of a file name. Since then, I have made individual folders and shortened the file names of those files I really need to get to.
Guess what. It still says, to long a file name. Drive E: \WIPB\Draft01.txt and \WIPB\Draft02.txt both still show too long a file name.
I'm totally baffled.
However, I found another way around the problem, that works.
I take the files Draft01.txt and Draft02.txt and upload them to Dropbox from my desktop in Debian.
Go to the notebook, boot into Win7, using WiFi to update my Dropbox folder. I create a new folder on the desktop, name it WIPC, open it, then copy Draft01.txt and Draft02.txt from Dropbox into the WIPC folder.
Now it opens, and I can read it, work on it, and save it.

Then out of curiosity, I did something as a test. I copied the Draft01.txt folder into the WIPB folder. It says file Draft01.txt already exists. Overwrite or Cancel. I hit Overwrite and it writes the file Draft01.txt into folder WIPB.
When I try to open the Draft01.txt from folder WIPB, I get the error message, filename too long.
OK, rename folder WIPB to WIPD. Open it by double clicking, click on Draft01.txt I get PATH NOT FOUND, hi hi.....
Aaargh.........

Last night, I created a single folder, in my shared folder in Debian with no subfolders, simply named WORK.
I took each of the 17 problematic files and renamed them using only 4 letters, plus the txt or odt extension.
Uploaded the folder WORK to Dropbox from my Debian machine.
Went to the notebook opened Win7, connected to Dropbox and let it update.
Copied the WORK folder from Dropbox to my Desktop.
Opened each file, changed a couple of words and resaved the files.
Opened them again using Notepad and Wordpad, not making any changes. They opened just fine.
Rebooted into Win7 and they were still there and working. I could open each, make changes to each and save each.
So I think I finally got around the filename too long problem.
I deleted all of the other files I had on my desktop and on the Drive E.
Then copied them all down fresh from Dropbox. The same 17 that dealt me the fits, still did, so I hit the SKIP button.
I checked several of the major files, and they worked without a problem.
Now dig this. I copied the 17 files already on the desktop, in folder WORK, to where they belonged in the original folders, without changing the file name from the 4 letters.
Once again I'm getting filename too long on each of those 17 folders.
While I have files that open just fine that have over 20 letters in the file name.

I'm beginning to think it is something other than the error message being given.

I don't remember the code now, but some guy had me type a code into Terminal, print out the output, for each file, one from a working folder and one from the filename too long.
Now I can't possibly type the weird letters that Terminal displayed under the title "Operand Stack" But they were nothing alike for the two files, anywhere in the entire list. From -Top of Stack- to -Bottom of Stack-.
Another list, called "Execution Stack" actually made sense on the file it read OK and was simple //def //end -filestream- //exec //begin //exch //def //end
However, the one from the file it couldn't read was much longer //def //end $error /_length //undef $error /init //get handleerror //if $error /dostop //get //stop //if //stop and keeps doing that for a page before changing to //_disableinterrupts //@aborted //_disableinterrupts (defaulttimeouts) //_extinfo //exec, this goes on for like four pages.

They guy had me type something else that used the previous Stack message files and it gave him the data he wanted to see.
Error Name: syntaxerror - Offending operator: (here is string of letters and symbols I can't type), then it gave
Error info string: --nostringval--

He had me e-mail him all of the saved files from our work on Terminal, saying, Yep, it's a bug alright.

He said, although I'm seeing the error message "File name too long," it is being generated because it is looking for a file name it cannot find, or one that is reporting "no_value." Which could explain the "Path not found" error message also.

How to fix it. Start over by making all new folders, including sub-folders, then copying my documents one by one into their corresponding folders. He suggested I do this using the DOCUMENTS folder, then moving my main folder to wherever I want it after I'm done. He said something about Nested Folders having a link to its parent folder, and somehow, somewhere, I have a link that is either corrupt, appended or verbose. He knew of no way to find exactly which one it was. And it is probably NOT the one that contains the unreadable documents.

He sorta made sense. I've had Simlinks go haywire in Ubuntu in the past. What is usually a short code, can sometimes be four lines long, for no apparent reason.

Technically, shouldn't a file name (or full path name actually) be able to be 32767 characters long on a 32 bit machine? And heaven knows how long on a 64 bit or 128 bit machine. Why is Win7 limited to 260 characters for the full path name? Look at some of the Images on Facebook with filenames that are miles long. It sees them!

TTUL
Gary
User avatar
Kellemora
Brainiac Class Poster
Brainiac Class Poster
 
Posts: 3389
Joined: 07 Jul 2012, 18:52
Location: From St. Louis, current Knoxville, TN

Re: Linux: Reading the Instructions

Postby Yogi » 12 Apr 2013, 14:54

I don't have an answer for you regarding file name conventions, but I will tell you what I know. First of all, from Microsoft itself, a Windows7 file name is indeed limited to 260 characters. The name includes the FULL PATH as part of those 260 characters.

What is the maximum length of a file name?
Windows usually limits file names to 260 characters. But the file name must actually be shorter than that, since the complete path (such as C:\Program Files\filename.txt) is included in this character count. This is why you might occasionally encounter an error when copying a file with a very long file name to a location that has a longer path than its current location.
http://windows.microsoft.com/en-us/wind ... -questions


Unlike what you see in a Linux /home directory, the Windows equivalent to /Documents, /Downloads, /Desktop, etc. is more complex. A user profile typically has something called /My Documents, /Downloads, /Desktop, and /Links as well as other folders for video, pictures, music, etc. Looking into the /Links directory in your profile, you will note shortcuts (links) to /Desktop, /Downloads, /Recent Places, and perhaps others. Thus, the path name to the Desktop you use to compose and store files can have a lot of hidden references. This is exactly what Linux is doing with symbolic links. You may see something like this when you do a long listing

    lrwxrwxrwx 1 root root 29 Apr 9 11:03 vmlinuz -> boot/vmlinuz-3.5.0-27-generic
This says that vmlinuz is actually not in the root directory from which you got the listing, but is instead located at boot/vmlinuz-3.5.0-27-generic. All those extra path characters become part of the file name via the symbolic link function. So it is with Wiindows when it comes to locating the /Desktop for example.

My approach is to not use the Windows provided library folders, but to create my own folders in which to store things like documents and downloads. My working desktop has folders called Desktop Downloads and Notes and Links whose path is not through the windows /Links directory, thus a lot shorter. In other words, the address to my created folders is relative to my desktop location, not an absolute address relative to the root directory. I can't be certain but I think the idea behind this symbolic linking in Windows is to make file sharing on a LAN easier. By appending the long path name anyone in the homegroup can locate a given file on your computer because it's an absolute address, not a relative one.

Thus, when you send something to your NAS, or The Cloud, you only see the symbolic link and not he entire absolute address. Depending on the origin of the document, the length of the path name changes even though the file name stays the same.

The length of a file name is not limited by the width of your data bus - 32 bits for example. A byte of data in a 32-bit system is 32 bits regardless. Each ASCII character is represented by 32 ones and zeros, but the number of those characters that can be used in a file name does not depend on the bit count. The limit is a function of the file system (NTFS) and the API used by Microsoft. I know this might be too much for you simply due to it's length and technical depth, but here is an excellent resource from Microsoft that explains in detail how the various naming conventions work: http://msdn.microsoft.com/en-us/library ... 47(v=vs.85).aspx
User avatar
Yogi
Oracle Class Poster
Oracle Class Poster
 
Posts: 7013
Joined: 04 Aug 2007, 19:37
Location: Chicagoland

Re: Linux: Reading the Instructions

Postby Kellemora » 13 Apr 2013, 09:13

Thanks Yogi

Had to hunt around a bit, because the link, although the right page, didn't have the articles. But I found them.

And it did help. I now have all of my too long filename problem solved.
I may not have understood it completely, but placing files on Drive E extends the file system, rather than shortens it.
To get to Drive E, from Win running from Drive C it goes to the Windows Directory on C, to the System 32 folder, back to the Windows folder, to the Documents and Setting folder, and is then directed to Drive E, then to the top folder on Drive E. etc.

By taking the file OFF the Desktop and moving it directly into the Documents and Setting folder. I no longer get the too long errors. Moving it to Drive C still produces the error. So apparently, the path from Win7 in the Windows folder to the Documents and Settings folder is shorter than going to Drive C's Root folder?

Also, making all new folders from scratch on the desktop worked. It was time consuming, but worked.

The thing that has me most puzzled is: If I download an image, with a mile long file name, from like Farcebook or YouTube, it opens with no problems from my desktop or any folder I stick it into. That filename by itself is often over 100 characters long.

I wonder if there is a filename limit in Linux? I'm sure there must be.
And I know I can't compare to what is done at work, because I think their system is technically considered a mainframe. Although in actuality, it is just a zillion PC boards all clustered together.

On my OWN PC, I can look at /root, which shows /home, then /gary, then all of my files.
At Work, I can only back up two more levels than my own named folder.
This is because I have access to certain working program files that affect other users. In fact we all do! But that does not mean I can get into other users folders. I have no idea how far above that folder lies /home, hi hi.....
However, I can back up far enough to see various departments. I cannot get into most of them, permission denied, hi hi.....
Have no idea how or why they named some folders what they did. But from /rhasim, the highest folder I can back up to. I see /editing, /kkr, /dprogs (which is where we get to program data), then to get back to my station I go through /depgusr (department g users, which shows me everyone's folder in my department, then of course /ihgary which is my folder.

We have several programs we can use that are NOT linked to our folder, nor or we allowed to add links to them. EG: place them in our start menu. Each time we use one of these programs, we have to tell it where to save the output file, if it lets us, some output files are fixed to the general file folders area, where everyone can get to them.
However, although we cannot make a link in OUR start menu, we learned a trick. We CAN add to our start menu an existing menu called systems applications (similar to going to Applications, Edit Menu's and selecting Debian as an application). All of the system programs are in this Systems Applications menu and we can uncheck those we cannot use and leave the ones we use often. Someone checked and it is OK for us to do it that way, since we cannot alter the main system menu from home or a workstation.

The only example I can give of how this is possible, since I work from home on my own computer, is to use Dropbox.
On my computer, besides appearing on my upper Panel as an Icon, Dropbox appears in my Applications/Internet drop down, as well as in my Applications/Debian/Applications/DataManagement drop down.

TTUL
Gary
User avatar
Kellemora
Brainiac Class Poster
Brainiac Class Poster
 
Posts: 3389
Joined: 07 Jul 2012, 18:52
Location: From St. Louis, current Knoxville, TN

Re: Linux: Reading the Instructions

Postby Yogi » 13 Apr 2013, 12:23

I understand what Microsoft is saying about length of file names. It's something I've run into since the early days of computers. I don't know what determines the limits, but I do know the path to a file is parsed by the operating system. The parser can only go so far by design. Both Microsoft Windows and Linux have been around for a long time. Their kernels are basically unchanged for decades. Microsoft did change with Windows 8, however. Because OS programmers try to cover every possible permutation of hardware and software in the wild, they do some strange things to remain compatible over the years. I think this appending of path names is one of those attempts at backward compatibility. In any event, I'm glad you found a solution to a sticky problem.

Dropbox is Cloud Computing at its finest. Given that it's web based I don't see a problem ever occurring with file name lengths. In fact the best possible world would keep all your data in a universally accessible storage device such as Dropbox. I realize there are other issues preventing that, but keeping all your work off line and independent of your work stations would simplify things tremendously.

Your employer has many more considerations than you do when it comes to security and file sharing. I'm sure they have an entire staff of database specialists managing group permissions and directory structures. It's more than you have to do at your home office fortunately.
User avatar
Yogi
Oracle Class Poster
Oracle Class Poster
 
Posts: 7013
Joined: 04 Aug 2007, 19:37
Location: Chicagoland

Re: Linux: Reading the Instructions

Postby Kellemora » 14 Apr 2013, 11:51

I'll keep this one super short today, with only a single, but not so simple question.

How does Dropbox, or any BIG web site for that matter, keep from losing their clients data?
I'm thinking also of on-line games, like Farm Town or Ravenwood Fair, played through Farcebook.
With over 6 million daily players, each making hundreds of changes to their files each day, and the workload split up between 40 or 50 separate servers around the country, all interacting with each other.
EG: I may be on server 16, but just visited a Farm on server 38, while players from servers 22, 28, 47 are all there at the same time. Each of these players data, the results of their actions, are stored and saved.

Maybe the latter is too much to answer, so for now, lets just leave it at a single but large web site with lets day 500 thousand clients. Or a site like Dropbox. How do they protect their clients data from loss?

As an aside: One reason I would use a HOST, rather than run my own servers or lease server space, is simply because I still have not learned HOW to ensure no data loss. Also WHY I do so much redundant backup of my files.

I do have a slow as molasses NAS. Two Drives, Hot Swap. If one drive fails, and I replace the failed drive, it is supposed to rebuild with no data loss. However, if the controller card fries. I've lost everything! So the NAS is not my main file server.

TTUL
Gary
User avatar
Kellemora
Brainiac Class Poster
Brainiac Class Poster
 
Posts: 3389
Joined: 07 Jul 2012, 18:52
Location: From St. Louis, current Knoxville, TN

Re: Linux: Reading the Instructions

Postby Yogi » 14 Apr 2013, 14:22

The short answer is that I don't know how Dropbox protects its clients data. I'd guess they do something like I outlined for you in another thread. Today's back-ups are on one server, that one is backed up by a weekly server, that one backed up by a monthly server, which is backed up by a quarterly server, etc, etc, etc.. Thus if one backup server goes down, there are a few other places they could look for data.

There is no way to protect all your data. The interval from your current session back to the last backup is vulnerable. That vulnerability could be partially resolved automatically by using a mirror server which updates as your workstation changes in real time. I doubt that Dropbox, or anybody not on a mission critical system, does that type of thing.

Multi-player games are typically done on a server installed on your computer. Steam or Hamachi, for example, use virtual networks to connect their players. In other words, each player is the hub for a network of all the players he encounters, who are in turn branches of his network. Updates for the individual players go to their home servers and are called into the virtual network(s) as needed.

I'm sure Farcebook does not follow convention and has their own way of doing things. But it's reasonable to expect that a virtual network of one kind or another is involved.

Simple, eh? :lol:
User avatar
Yogi
Oracle Class Poster
Oracle Class Poster
 
Posts: 7013
Joined: 04 Aug 2007, 19:37
Location: Chicagoland

Re: Linux: Reading the Instructions

Postby Kellemora » 15 Apr 2013, 11:41

Hi Yogi

I've been talking to my dealer about a new computer, and the methods of back (mirroring) he recommends is logical, but not practical for most of my work.

Although he sells tons of RAID arrays, he emphatically states that there is no reason for a home user, even someone running a small business, to mess with them.

He knows my system and how I currently do things, and says that I can't get much safer than that. Although he would add DVE to my system, preferably BlueRay for archival storage of non-changing data.

Simply because I use more than one computer, using the internal HD's would be more confusing that using a dedicated external drive for all the data, and mirroring it off-site as I do.

He has a software program for Windows computers that watches and tracks your data, making copies to a BlueRay machine for files that have not changed. I didn't explain it very well, because I run Linux and Rsync does basically the same thing.

What I used to do, and what I do now, since getting the NAS, is only a slight variance.
I made a copy of my external HD I use as my file server to the NAS, so all the data is available to me.
I normally NOW, keep the external HD turned off. Which also means my automated nightly mirror does not take place.
I keep folders on my Desktop for my daily work, save them to the NAS, and once a week turn on the External and copy them to that. Then leave it on, so Rsync will mirror it to my other external down at the house, and then to my out-state HD at my brothers. Rsync is great in that it only copies changed files, not everything, which takes forever.
Even so, I still have to do a full mirror from time to time, into a different folder, as a safety measure against a corrupt file.

That is one of the downfalls of automation. A file that was good and backed up to other drives, becomes corrupt.
So when the next backup is made, that overwrites the existing files. The good working copy gets overwritten with the corrupt file. Thus the reason to have archived backups.

He said RAID is normally only needed when you need more speed, for many users and connects to the server data.
For a single user like myself, If I really needed more speed, I could use an internal SD drive, but not to trust them for storage.
I would use it like I use files I keep on my desktop.

One last thing. I asked him if it would be safer to used linked files. A folder on my desktop is LIVE with the folder on the external HD. He said NO do not do that at all. I would be much safer simply leaving the mounted volume on my desktop, than creating links from different folders on different computers. The way he explained it, I could see how making a change on one computer, then moving to another, could cause corruption. Linked folders are not updated to match the source.

FWIW: He himself has purchased storage space, like using the cloud, only it's with a private company. But that does not preclude his keeping his own backups of everything.

TTUL
Gary
User avatar
Kellemora
Brainiac Class Poster
Brainiac Class Poster
 
Posts: 3389
Joined: 07 Jul 2012, 18:52
Location: From St. Louis, current Knoxville, TN


Return to Personal Computers

Who is online

Users browsing this forum: No registered users and 2 guests

cron