remote fileserver mountable as filesystem. how ?

[free_electron]

i know this is not exactly stuff for eevblog but i got no clue where to ask this.

here is what i want :
- a fileserver that is stored somewhere. don't care where.
- connected to the internet with a static IP address
- password protectected with user accounts.
- automatic backup with daily and weekly snapshots.
- manageable ( meaning i can set up folders create users and assign permissions to folders for users. )

- mountable as a drive . No 3rd part software , no ftp , no synctoys. on windows : my computer -> map network drive -> done. from anywhere in the world  <-tricky bit

here is the problem: i have a set of files i (and other people) want to work on no matter where we are in the world.
just like having a network mounted NAS at home i want an internet mounted NAS. all nas devices can point toward internet but they all use a web based interface to transfer files or ftp or some sync method. that is NOT what i want. i need mounting as a drive letter. if a file is saved it needs to go directly to the remote machine without local copy. (local copy means syncing is in order ... no syncing ! )

i tried stuff like dropbox, google drive and others and they don't work for me. google drive just fucked up royally by deleting a bunch of master files during a sync. apparently when you move stuff into the folder without the drive app running the sync goes haywire...
i don't want to install 3rd party software either and no workarounds with ftp , synctoys or anything. i do not trust synchronisation. if sync goes wrong you lose the latest version (different timezones etc )

all i want is go to my  file explorer : map network drive : point it to an ip address and  kablam. done. just like i mount a local nas
so that, when i open a fie in a program and click 'save' it is directly written to the remote storage. no local copies needed.

i looked for a few hours on google but can;t find really a solution.

[n45048]

Windows will allow you to map a drive to a network location over the internet (via SMB, FTP or HTTP), but I would strongly recommend that you don't have SMB shares open to the internet. You're risking the security of your data.

Ideally what you should look at is a VPN based solution and map your drives over a secure network.

[free_electron]

i found vpn but what hosting provider supports that. i tried godaddy and others and its njet ..
i don't want to run my own server. it's gotta sit a some hosting provider,  be backed up , duplicated and have 99.999% uptime ..

[n45048]

i found vpn but what hosting provider supports that. i tried godaddy and others and its njet ..
i don't want to run my own server. it's gotta sit a some hosting provider,  be backed up , duplicated and have 99.999% uptime ..

That I can't help you with off the top of my head. I don't use external providers for anything other than domain hosting. You'd have to do your own homework there. I run all my own servers.

[apelly]

I assume you use windows.

You are looking for WebDAV.

For the client the setup is similar to your outline.

There are open source options, but I think would prefer something paid/with support.

Microsoft sharepoint is an option.

[Lightages]

I used this a long time in the past and it worked well:
http://www.netdrive.net/

Also just found this too.
https://www.eldos.com/sftp-net-drive/download-release.php

[timb]

free_electron; send me a PM and I'll hook you up with exactly what you're looking for!

I spent 15 years in IT setting up and managing custom servers for companies, might as well put some of my knowledge back to use!


Sent from my Smartphone

[free_electron]

I used this a long time in the past and it worked well:
http://www.netdrive.net/

Also just found this too.
https://www.eldos.com/sftp-net-drive/download-release.php

i found netdrive as well. but i am opposed to having to use a third party tool. the problem is some clients are on macos , some on linux, some on windows. i really need to be able to simply map a network drive.

just like in my home i can see my nas boxes automatically in the network browsers of my windows and mac machines and they become an extension of the file system. no 3rd party software needed.

icloud, google drive , skydrive etc don't work right. you have no 'direct' access to the actual file system. you always need to save locally and then depend on the file sync. It needs to be transparent for any software or operating system we use: it needs to behave like a real drive / folder does, just like when you mount a share as a drive letter. Just in this case the share does not sit on the local lan but on the internet.

An encrypted link would be nice.  VPN comes to mind. but that is another can of worms....

WebDAV seems to be a solution. i did not know that existed. ( i'm not a computer guru ). Looks like it is a standard too.
Now to find a provider that has webdav in his offerings and has  an easy management console to set up folders / users and assign permissions.

-edit- hmmm webdav does nto seem to be what i am looking for. it does not seem to have file permissions or user permissions. once you login to the webdav you have full control over everything there. that is not what i want.

the scenario is that we are a group of people that will each have their own user folder. there will be projects folders. only people assigned to that project have write permissions there, otheres are read only. etc. so it really needs to behave like a local file server inside a company network would. including all the permission settings.

growl... why are such simple things so damn hard.

i am trying to get a group of people ,spread over different locations, work together just like as they were sitting in the same building / office and have a local fileserver. VPN tunnels seem to be made for that. but i got no clue if this can be set up at a provider ... i know amazon EC2 allows VPS virtual private servers where you can deploy a server image yadda yadda.. i don't want to do that for various reasons :

1 - i don't want to be responsable for configging the server. all i need is an internet facing nas box: plug and play.
2 - you pay by the hour of that thing running .. woops... too expensive.

[ovnr]

Apache's DAV implementation should do what you want. You can set up per-directory access limits (users or user groups) without too much trouble. You can probably just drop .htaccess files into each project dir to set up user groups.

(Disclaimer: I haven't actually tried it. Should work tho.)

[nctnico]

growl... why are such simple things so damn hard.

i am trying to get a group of people ,spread over different locations, work together just like as they were sitting in the same building / office and have a local fileserver. VPN tunnels seem to be made for that. but i got no clue if this can be set up at a provider ... i know amazon EC2 allows VPS virtual private servers where you can deploy a server image yadda yadda.. i don't want to do that for various reasons :

You are going the wrong way with this. What you need is a version control system. When working with multiple people on the same project you can't simply share files. You need a way to tell who changed what, merge files and go back to previous versions etc.

[geppa.dee]

What you're describing is SMB over VPN. No other way around it (that I know of). I don't know about hosters providing directly that but you can sure do better than hourly paying Amazon. I use Linode.com for a bunch of virtual servers and even their lowest offering would serve you well and cost you very little. Someone would have to config the VPS for you initially though. Not something that can't be done in a couple of hours...

OTOH... you're sure that's what you want? What if you have that "drive" mounted and a file (or two) from it opened and... you loose the internet connection? In the local LAN... that's unlikely but over the internet... not so much.
IMO, nctnico's suggestion for version control with local checkouts if far better...
No way to loose data anyway you put it.

[miguelvp]

Old school nfs (network file system) or sshfs (SSH file system)?

nfs is usually blocked by firewalls/routers/etc sshfs is not.

Not sure if there is a way to do this in windows though

[Psi]

Dedicated vpn boxes at all locations to link the sites.
 
done. done.

[free_electron]

growl... why are such simple things so damn hard.

i am trying to get a group of people ,spread over different locations, work together just like as they were sitting in the same building / office and have a local fileserver. VPN tunnels seem to be made for that. but i got no clue if this can be set up at a provider ... i know amazon EC2 allows VPS virtual private servers where you can deploy a server image yadda yadda.. i don't want to do that for various reasons :

You are going the wrong way with this. What you need is a version control system. When working with multiple people on the same project you can't simply share files. You need a way to tell who changed what, merge files and go back to previous versions etc.

Version control system is useless. They are binary files... Not sourcecode. Cad drawings, mechanical ,electrical . Each person has a task assigned and owns filesets. I deliver the pcb files, and a step file. Our mechanical dude runs solidworks . We exhange drawings. It's not sourcecode. There are no two person working on the same file. Ownership is cut and dried.

Smd through vpn seems the solution. Dedicated vpn boxes.. Thats gonna cost money. Money we dont have .. Running in startup mode you know .. Pre investor ...

-edit- found an interesting document from cisco
http://www.cisco.com/c/dam/en/us/td/docs/routers/csbr/app_notes/rv0xx_g2gvpn_an_OL-26286.pdf

Mesh vpn. Then we don't need a central server. Or, i can simply get a static ip address, slap a NAS with dual drives on my network and off we go. No server needed.

Looks like vpn routers are not that expensive... Hmm homework to do ...

[ovnr]

Version control systems work perfectly well for binary data as long as you don't have any merge conflicts. If you can get everyone to use it, there should be no problems. Plus you get, you know, version control.

It'd also be significantly easier to set up and deal with than a shared drive - and likely cheaper, all you need is a VPS somewhere or a managed account from someone else. VPSes start at $5-$10/month; you can even get a dedicated server for like $30/month these days. (I've got one - unlimited 100mbit line, 1TB disk, acceptable performance. I pay $29.95/month.)

[nctnico]

Indeed. Version control systems also work perfectly for binary data. Deleted something from a drawing? Check-out the previous version and copy&paste.

[free_electron]

Indeed. Version control systems also work perfectly for binary data. Deleted something from a drawing? Check-out the previous version and copy&paste.

maybe i'm retarded when it comes to these things but explain me how a version control system would work betwween for example 2 schematic files. i can't see how it can tell me: you removed R3 and added C5 . not unless it understands the internal structure of those schematic files.

that kind of stuff works for sourcecode : added a line here, removed two lines there. scroll side by side. for binary files  all it can do is tell you they were different. it cannot tell you what was changed. not unless it knows internal file layout. and even then ...

unless it can do the above ( tell me what parts were removed and or altered) it is useless for that kind of work. (altium creates ECO (engineering change order) files telling me what was changed. whenever i create a mod of a schematic a revision number is slapped on. ECO's are saved . so i can go back. no need for extra software to do that. i can simply browse the eco's. and pick up the file associated.

same for pcb .

once a design is a release candidate then RC is added to the file name

for example
Fubar-1
fubar-2
fubar-3
fubar-4
fubar-4-rc1
fubar-4-rc2
fubar-4-release
fubar-5
fubar-6
fubar-6-rc1
fubar-6-release

and so on

the releases are frozen in time fubar-5 would be the successor to fubar-4-release. it;s very simple

[edavid]

Indeed. Version control systems also work perfectly for binary data. Deleted something from a drawing? Check-out the previous version and copy&paste.

maybe i'm retarded when it comes to these things but explain me how a version control system would work betwween for example 2 schematic files. i can't see how it can tell me: you removed R3 and added C5 . not unless it understands the internal structure of those schematic files.

It can't give you change information.

all it can do is tell you they were different. it cannot tell you what was changed.

Yes, everyone knows this.

unless it can do the above ( tell me what parts were removed and or altered) it is useless for that kind of work.

This is where you are wrong.  It can still take the place of a distributed filesystem/shared drive, but will be much easier to set up, more reliable, and have better performance.  And, you get versioning for free.

[miguelvp]

If it's source control (or binaries for that matter) perforce has a free 20 seat version with unlimited size and no time constrains.

Just google perforce 20 seat

One thing is that you have to check out the files before working on it unless your software has built in source control and perforce support.

After you are done you can revert unchanged files with a single right click on the change list, add comments on the work done and check it in.

You can compare any file with any previous versions, hook your favourite file compare software (beyond compare is pretty good) or use the built in one.

You can revert changes, label versions, branch to other projects from a common one, well a lot of things but it might take you a bit to ramp up to understand how it work, but they have plenty of tutorials and more are available in the interwebs.

[free_electron]

It can't give you change information.

maybe my understanding of a version control tool is wrong , but : if it can't do that then it is useless.

We had a version control system called Rational Clearcase used for collaborative code design. it could show you exactly who had changed what line of coade, what that line was at any given point in time, what lines were added/removed between versions etc. so for sourcecode this is a great system.
but for any other kind of filetype : useless because it does not understand what is there. all it can tell you is the files are different. big whoop. a diff command under dos can do that .. don't need the overhead of another tool to learn .

This is where you are wrong.  It can still take the place of a distributed filesystem/shared drive, but will be much easier to set up, more reliable, and have better performance.  And, you get versioning for free.

explain this with an example. i have trouble grasping this concept. ( see above. )
here is my experience with this kind of stuff : we tried using google driv.e that supposedly has versioing built in. good effing luck. you got no cluse waht version you are opening, it messs up syncing and keeps converting my .pcbdoc or schdoc files to .doc extention when downloading them beacuse bloody google drive things any file ending in doc is a word document... they can't even properly parse file extensions. all they look for is the last 3 characters.. they should parse for the characters after the last dot in the filename. So i am very reluctant to go that path again ... too much hassle

anyway, we are deviating. the idea is to run SMB over the internet (wan) through vpn. i don't want intermediate storage, syncing or any other crap. it has to be transparent for the programs we use. so it needs to be visible as a filesystem. Hit save and the master copy on the fileserver is updated in realtime.

[edavid]

here is my experience with this kind of stuff : we tried using google driv.e that supposedly has versioing built in. good effing luck. you got no cluse waht version you are opening, it messs up syncing and keeps converting my .pcbdoc or schdoc files to .doc extention when downloading them beacuse bloody google drive things any file ending in doc is a word document... they can't even properly parse file extensions. all they look for is the last 3 characters.. they should parse for the characters after the last dot in the filename. So i am very reluctant to go that path again ... too much hassle

That was the wrong tool, too bad you had a bad experience, but forget it.  Look at git instead, or maybe someone else has a different suggestion.

anyway, we are deviating. the idea is to run SMB over the internet (wan) through vpn. i don't want intermediate storage, syncing or any other crap. it has to be transparent for the programs we use. so it needs to be visible as a filesystem. Hit save and the master copy on the fileserver is updated in realtime.

I know that's the idea, unfortunately it's a bad idea.  It seems attractive, but it will never be transparent, because the Internet is in the way.  Take a step back and look at the real problem, which is distributed project development.  This is not exactly new.  The best solution that's available now is a version control system.

[nctnico]

Indeed. Version control systems also work perfectly for binary data. Deleted something from a drawing? Check-out the previous version and copy&paste.

maybe i'm retarded when it comes to these things but explain me how a version control system would work betwween for example 2 schematic files. i can't see how it can tell me: you removed R3 and added C5 . not unless it understands the internal structure of those schematic files.

This is why you enter a remark during the commit. This would say: removed R3, added C5 to improve stability yadda yadda.
With sources / text files a version control system can show the difference between the source code but that usually is of little value if the reason why is unknown. The commit remark should contain information on why (fix which issue) and how the change was made or whether it is just an intermediate commit to save work.

[mrflibble]

anyway, we are deviating. the idea is to run SMB over the internet (wan) through vpn. i don't want intermediate storage, syncing or any other crap. it has to be transparent for the programs we use. so it needs to be visible as a filesystem. Hit save and the master copy on the fileserver is updated in realtime.

Holy fucking shit. I seriously laughed out loud. 

Please don't do that, you are going to hate yourself. SMB over vpn. Just google around a bit for issues regarding that. Smb isn't the best of protocols for the problem you are trying to solve. Honestly a version control system really sounds like a decent fit. Even with the binaries in the mix. For example Altium works nicely with subversion. You get your diffs same as you are used to now, AND you get easy to read commit history from you and other people. Find a specific version before you started messing up some pcb area? No more difficult than what you are used to now.

And you can make tags for major revisions. Want revision XYZ of board? Just do a subversion checkout of that revision tag. Personally I find that easier to use than the local file based method. That and it's automatically backed up. But then again, I would assume your altium projects are also on a network share with backup. Would be a bit silly not to.

Git works with altium files just as well, except AFAIK Altium does not have builtin git support. So you'd use a git client in addition to altium.

[miguelvp]

A simple example on my pretty virgin P4V install. (Edit: this is the free 20 seats one, unlimited size and no time expiration or anything like that)

Note that when collaborating if people make changes on a file you are working on as well it does allow you to resolve changes, but if they are binary you might want to exclusive check them out so no one can alter them and they have to wait until you are done so then can start adding their changes after your's are in the depot.

But usually everything I do is text based so.

This is the PSoC KitProg project that I modified just a couple of i/o pins and change the version number so it's compatible with current cypress tools.


First I did check in the full project as I downloaded it and intended for Creator V2.0.
Then I proceeded to clean temp files that really don't belong on source control.
Then I fired up Creator 3.0 and allowed the program to upgrade my modules, I added notes on modules I didn't let the software upgrade.
Then I changed the programming pins.
And last I upgraded the version of KitProg because current cypress tools expect 2.08 and will refuse to use my custom 2.03 even if they are compatible.

You can open a changelist and it will show you what files changed.


And you can diff them as well


Many other things too. And even if I use this locally, the depot can be in the cloud. Hopefully your datacenter will make backups or you will have to do that, just incremental backups will work and doing full ones every now and then.

If many branches and users, you can do revision graphs to see how a piece of code changed all the way to it's original inception.


Edit 2:

Also Perforce has an API so you can do your own extensions if you are so inclined to.
http://www.perforce.com/product/components/apis

20 seat download page:
http://www.perforce.com/downloads/Perforce/20-User

Supports Unix, Windows, Linux, and Macs.
And it's Git compatible to some degree, haven't used that yet but pretty much it can show your depot as a Git, or is it the other way around? not sure but you can read the details here:
http://www.perforce.com/git-fusion

Also it has peer review capabilities via their Perforce Swarm.

[ovnr]

It can't give you change information.

maybe my understanding of a version control tool is wrong , but : if it can't do that then it is useless.

But you want a method to distribute project files, not a VCS. If the VCS doesn't provide changelogs like you'd want it to, so what? It's not like you'd have that if there was a shared drive either.

The only difference for your application is that changes won't appear magically; you'd have to manually refresh it, as well as manually check it into the repository. Plus it stores backups of all previous versions, so if you go "Gah! I deleted these parts last week from the schematic, and now it turns it they were needed. Wonder what values they were?", you can just check out the old version and look it up. Don't get sidetracked by trying to get merges working or anything.


And the reason it gets sidetracked is because a shared drive over the internets is a pretty horrible idea (on top of shared drives being a pretty horrible idea even on a local network, at least when it comes to project management...). If we dive right into the root of the problem: You have a set of people who collaborate on a project, and thus need to have up-to-date copies of all project data. The logical conclusion is not "I know, I'll set up a shared drive and party like it's 1999!", but "I'll pick a decent collaboration tool".


Also, if you've never run a VPN, it can be absolutely fascinating. The amount of problems they can generate is hilarious.