发信人: sandbird (sandbird), 信区: Statistics
标 题: 能不能让R 读一个 150G 的 table?
发信站: BBS 未名空间站 (Fri Jan 30 18:10:32 2009)

Can I ask R to "read.csv" of a 150G file and extract records from this table
.

Anyone has similar experience doing this in Windows 64bit machine?

or any suggestions of doing it under Unix R or Windows R?

Thanks
--

※ 来源:·WWW 未名空间站 海外: mitbbs.com 中国: mitbbs.cn·[FROM: 128.231.]

发信人: hezhi (荷芝), 信区: Statistics
标 题: Re: 能不能让R 读一个 150G 的 table?
发信站: BBS 未名空间站 (Fri Jan 30 19:48:28 2009), 转信

太强了吧
【 在 sandbird (sandbird) 的大作中提到: 】
: Can I ask R to "read.csv" of a 150G file and extract records from this
table
: .
: Anyone has similar experience doing this in Windows 64bit machine?
: or any suggestions of doing it under Unix R or Windows R?
: Thanks



--

※ 来源:·BBS 未名空间站 海外: mitbbs.com 中国: mitbbs.cn·[FROM: 98.212.]

发信人: leohawk (leohawk), 信区: Statistics
标 题: Re: 能不能让R 读一个 150G 的 table?
发信站: BBS 未名空间站 (Fri Jan 30 20:14:13 2009), 转信

kao, niu a ...
if you know what you want, better use grep to pre-process ba...
I do not think you can do that on even a super computer

【 在 sandbird (sandbird) 的大作中提到: 】
: Can I ask R to "read.csv" of a 150G file and extract records from this
table
: .
: Anyone has similar experience doing this in Windows 64bit machine?
: or any suggestions of doing it under Unix R or Windows R?
: Thanks



--

※ 来源:·BBS 未名空间站 海外: mitbbs.com 中国: mitbbs.cn·[FROM: 71.251.]

发信人: RProgrammer (RP), 信区: Statistics
标 题: Re: 能不能让R 读一个 150G 的 table?
发信站: BBS 未名空间站 (Fri Jan 30 22:42:25 2009)

Almost impossible.

So far as I know, there is no stable version of 64-bit version of R under
windows, and if I remembered correctly,u nder windows, the maximum memory of
R
that can use is 2 G.

For Unix or Linux, there are 64-bit version of R, but the maximum memory
space
R can utilize is still limited, say, some package can not handle more than a
certain number of entries (2^(32)-1) in a matrix, or an object can not be
more
than a certain size that C or Fortran can handle, no matter how large your
machine's physical memory is.

Maybe you can do a feature selection/dimension reduction, etc. to bring down
the volume of the data, or do all the calculation using low level
programming
language other than R, or probably do all the calculation in the database
using
the built in database syntax, or select a portion of the data samples to
load
into R and do it iteratively if it is appropriate for your calculation.




【 在 sandbird (sandbird) 的大作中提到: 】
: Can I ask R to "read.csv" of a 150G file and extract records from this
table
: .
: Anyone has similar experience doing this in Windows 64bit machine?
: or any suggestions of doing it under Unix R or Windows R?
: Thanks




--
各有所好吗～～～

※ 来源:·WWW 未名空间站 海外: mitbbs.com 中国: mitbbs.cn·[FROM: 70.187.]

发信人: leftHand (Hold on...), 信区: Statistics
标 题: Re: 能不能让R 读一个 150G 的 table?
发信站: BBS 未名空间站 (Sat Jan 31 13:46:40 2009), 转信

is there a 64bit R for windows? is there a compile version
that can be downloaded?

【 在 RProgrammer (RP) 的大作中提到: 】
: 标 题: Re: 能不能让R 读一个 150G 的 table?
: 发信站: BBS 未名空间站 (Fri Jan 30 22:42:25 2009)
:
: Almost impossible.
:
: So far as I know, there is no stable version of 64-bit version of R under
: windows, and if I remembered correctly,u nder windows, the maximum memory
of
: R
: that can use is 2 G.
:
: For Unix or Linux, there are 64-bit version of R, but the maximum memory
: space
: R can utilize is still limited, say, some package can not handle more than
a
: certain number of entries (2^(32)-1) in a matrix, or an object can not be
: more
: than a certain size that C or Fortran can handle, no matter how large your
: machine's physical memory is.
:
: Maybe you can do a feature selection/dimension reduction, etc. to bring
down
: the volume of the data, or do all the calculation using low level
: programming
: language other than R, or probably do all the calculation in the database
: using
: the built in database syntax, or select a portion of the data samples to
: load
: into R and do it iteratively if it is appropriate for your calculation.
:
:
:
:
: 【 在 sandbird (sandbird) 的大作中提到: 】
: : Can I ask R to "read.csv" of a 150G file and extract records from this
: table
: : .
: : Anyone has similar experience doing this in Windows 64bit machine?
: : or any suggestions of doing it under Unix R or Windows R?
: : Thanks
:
:
:
:
: --
: 各有所好吗～～～
:
: ※ 来源:·WWW 未名空间站 海外: mitbbs.com 中国: mitbbs.cn·[FROM: 70.187.]



--

※ 来源:·BBS 未名空间站 海外: mitbbs.com 中国: mitbbs.cn·[FROM: 68.50.]

发信人: daydayup1 (咳咳), 信区: Statistics
标 题: Re: 能不能让R 读一个 150G 的 table?
发信站: BBS 未名空间站 (Sat Jan 31 15:33:37 2009)

那什么语言能干这种事情呢? SAS 行吗? MATLAB, S-PLUS?
【 在 RProgrammer (RP) 的大作中提到: 】
: Almost impossible.
: So far as I know, there is no stable version of 64-bit version of R under
: windows, and if I remembered correctly,u nder windows, the maximum memory
of
: R
: that can use is 2 G.
: For Unix or Linux, there are 64-bit version of R, but the maximum memory
: space
: R can utilize is still limited, say, some package can not handle more than
a
: certain number of entries (2^(32)-1) in a matrix, or an object can not be
: more
: ...................



--

※ 来源:·WWW 未名空间站 海外: mitbbs.com 中国: mitbbs.cn·[FROM: 67.80.]

发信人: sir ( 郎 ), 信区: Statistics
标 题: Re: 能不能让R 读一个 150G 的 table?
发信站: BBS 未名空间站 (Sat Jan 31 16:11:00 2009)

read line by line
read one, throw one
--

※ 来源:·WWW 未名空间站 海外: mitbbs.com 中国: mitbbs.cn·[FROM: 64.69.]

发信人: leohawk (leohawk), 信区: Statistics
标 题: Re: 能不能让R 读一个 150G 的 table?
发信站: BBS 未名空间站 (Sat Jan 31 16:32:29 2009), 转信

it is gonna awefully slow
【 在 sir ( 郎 ) 的大作中提到: 】
: read line by line
: read one, throw one



--

※ 来源:·BBS 未名空间站 海外: mitbbs.com 中国: mitbbs.cn·[FROM: 71.251.]

发信人: sir ( 郎 ), 信区: Statistics
标 题: Re: Re: 能不能让R 读一个 150G 的 table?
发信站: BBS 未名空间站 (Sat Jan 31 18:30:48 2009)

用时间换空间啦
--

※ 来源:·WWW 未名空间站 海外: mitbbs.com 中国: mitbbs.cn·[FROM: 71.233.]

发信人: RProgrammer (RP), 信区: Statistics
标 题: Re: 能不能让R 读一个 150G 的 table?
发信站: BBS 未名空间站 (Sat Jan 31 23:32:58 2009)

I remembered Brian Ripley said he had compiled one but crashes all the time.
I am not sure whether there is a stable compiled version ready to be
downloaded now.



【 在 leftHand (Hold on...) 的大作中提到: 】
: is there a 64bit R for windows? is there a compile version
: that can be downloaded?
: of
: a
: down



--
公司来了个老印...对我有些儿崇拜...

※ 来源:·WWW 未名空间站 海外: mitbbs.com 中国: mitbbs.cn·[FROM: 70.187.]

发信人: sandbird (sandbird), 信区: Statistics
标 题: Re: 能不能让R 读一个 150G 的 table?
发信站: BBS 未名空间站 (Sun Feb 1 01:00:07 2009)

thanks for all of your replies, though i still haven't found a solution yet.
--

※ 来源:·WWW 未名空间站 海外: mitbbs.com 中国: mitbbs.cn·[FROM: 76.114.]

发信人: NKVD (Chuikov), 信区: Statistics
标 题: Re: 能不能让R 读一个 150G 的 table?
发信站: BBS 未名空间站 (Sun Feb 1 02:07:05 2009), 转信

hadoop
【 在 sandbird (sandbird) 的大作中提到: 】
: Can I ask R to "read.csv" of a 150G file and extract records from this
table
: .
: Anyone has similar experience doing this in Windows 64bit machine?
: or any suggestions of doing it under Unix R or Windows R?
: Thanks


--
In dream: D700 + 300 F/4 VR.





※ 来源:·BBS 未名空间站 海外: mitbbs.com 中国: mitbbs.cn·[FROM: 76.87.]

发信人: leohawk (leohawk), 信区: Statistics
标 题: Re: 能不能让R 读一个 150G 的 table?
发信站: BBS 未名空间站 (Sun Feb 1 05:28:42 2009), 转信

I compiled R 2.7 on a 64 bit SLED, it took me a while to get
it done, mainly getting all required packages. I do not think
there are compiled version for download.

【 在 RProgrammer (RP) 的大作中提到: 】
: I remembered Brian Ripley said he had compiled one but crashes all the
time.
: I am not sure whether there is a stable compiled version ready to be
: downloaded now.



--

※ 来源:·BBS 未名空间站 海外: mitbbs.com 中国: mitbbs.cn·[FROM: 71.251.]

发信人: barrybarry (barrybarry), 信区: Statistics
标 题: Re: 能不能让R 读一个 150G 的 table?
发信站: BBS 未名空间站 (Sun Feb 1 12:34:16 2009)

Depending on the nature of your problem, you may try using the SCAN function
to read line-by-line and do the processing. Or, you can first load to data
into SQL database, and process only a portion of the whole data. The bottom
line is that R could not handle such a database all at once. SAS is better
suited for this kind of job.
--

※ 来源:·WWW 未名空间站 海外: mitbbs.com 中国: mitbbs.cn·[FROM: 76.108.]

发信人: statcompute (statcompute), 信区: Statistics
标 题: Re: 能不能让R 读一个 150G 的 table?
发信站: BBS 未名空间站 (Sun Feb 1 16:23:54 2009)

if i were you, i will:
1) read the data in to R piece by piece
2) insert cases into a light-weight database, such as slqite.
3) R has a very good interface with sqlite and you can easily extract data
from sqlite by R.
--

※ 来源:·WWW 未名空间站 海外: mitbbs.com 中国: mitbbs.cn·[FROM: 68.83.]

发信人: barrybarry (barrybarry), 信区: Statistics
标 题: Re: 能不能让R 读一个 150G 的 table?
发信站: BBS 未名空间站 (Fri Feb 6 10:25:51 2009)

try the "filehash" package.



--

※ 来源:·WWW 未名空间站 海外: mitbbs.com 中国: mitbbs.cn·[FROM: 208.62.]