Maybe there's something wrong with our config but the example command
trips an error
[root@hepgrid11 ~]# dpm-sql-usage-by-vo-user --es
Traceback (most recent call last):
File "/usr/bin/dpm-sql-usage-by-vo-user", line 99, in <module>
main()
File "/usr/bin/dpm-sql-usage-by-vo-user", line 66, in main
c = MySQLdb.connect(host=h, user=u, passwd=p, db='user_accounting')
NameError: global name 'h' is not defined
The command works fine without --es and --vo works fine as well.
John
On 15/10/2013 13:48, Sam Skipsey wrote:
> Right. So that's precisely what the tool in the toolkit is for.
>
> If you look at the file "usage_accounting", which should be installed
> in /usr/bin by the current version of the admin-tools rpm, then you'll
> see it's a sketch of a cron job which can be run daily to add the kind
> of usage information you're interested in to an additional database on
> your MySQL instance. While the visualisation layer is deprecated, it's
> not hard to query the records directly (and if there's interest, I can
> write the tools to do data presentation and searching for you).
>
> Sam
>
> On 14 October 2013 14:52, Elena Korolkova <[log in to unmask]> wrote:
>> Hi Sam
>>
>> I want to check how much space each VO was using for the last day. I don't keep records how much space each vo was using yesterday. I want to understand which vo is responsible for high disk usage. It difficult to see if I only have information on how much space each VO is using now.
>>
>> Thanks
>> Elena
>>
>> On 14 Oct 2013, at 13:27, Sam Skipsey wrote:
>>
>>> Hi Elena,
>>>
>>> I really just want you to clarify what you mean by "recent disk usage".
>>>
>>> If all you want it "how much space each VO is using now" then the
>>> toolkit has a tool for that. If you want the breakdown by
>>> diskserver/filesystem, then that's doable, but there's no tool for it
>>> at the moment.
>>>
>>> Sam
>>>
>>> On 14 October 2013 12:44, Elena Korolkova <[log in to unmask]> wrote:
>>>> Hi Sam
>>>>
>>>> I think it's better to ask snoplus to write their data to a space token (as we asked t2k).
>>>>
>>>> If there is no a simple tool to check a recent disk usage I can run ls-l command over file systems when I see a problem.
>>>>
>>>> Thanks
>>>> Elena
>>>>
>>>> On 14 Oct 2013, at 10:48, Sam Skipsey wrote:
>>>>
>>>>> Hi Elena,
>>>>>
>>>>> Of course, you could reserve a spacetoken for ops (but they'd have to
>>>>> actually try to use it)...
>>>>>
>>>>> As for the checking of recent disk usage - do you mean you'd like a
>>>>> historical view of how much disk has been used over time? That's hard
>>>>> to calculate (as the DPNS doesn't know about deleted files, and the
>>>>> DPM's transfer records of course get deleted for most sites on the
>>>>> timescale of 9 months or so). There's a tool in the dpm toolkit that
>>>>> is designed to be used to keep historical data like this, but it needs
>>>>> to be run as a cron job (and the visualisation tool is currently
>>>>> deprecated in favour of DPM Nagios and monitoring probes...)
>>>>>
>>>>> Sam
>>>>>
>>>>> On 13 October 2013 13:10, Elena Korolkova <[log in to unmask]> wrote:
>>>>>> Hi
>>>>>>
>>>>>> I was wrong about biomed. I found that this is snoplus. Just two files was put on Sheffield storage early morning but both are over 2 TB. It reminds me the problem we had with t2k. The problem was resolved when t2k set up a spacetoken.
>>>>>>
>>>>>> Elena
>>>>>>
>>>>>> On 13 Oct 2013, at 03:39, Elena Korolkova wrote:
>>>>>>
>>>>>>> Hello
>>>>>>>
>>>>>>> starting from Thursday ops nagios tests start to fail because there is no free space on the disk.
>>>>>>> I add 2 TB every day by decreasing atlas space tokens (mostly atlasproddisk) but on the next day all disk space is used again.
>>>>>>>
>>>>>>> After adding 2 TB today dpm-listspaces shows
>>>>>>>
>>>>>>> [root@lcgse0 ~]# dpm-listspaces
>>>>>>>
>>>>>>> POOLS:
>>>>>>>
>>>>>>> lcgPool
>>>>>>> CAPACITY: 327.89T RESERVED: 315.00T UNAVAIL (free/used): 10.90T/0.00
>>>>>>> USED: 669.58G FREE: 1.33T (0.4%)
>>>>>>> Space Tokens: ATLASDATADISK, ATLASLOCALGROUPDISK, ATLASPRODDISK,
>>>>>>> ATLASSCRATCHDISK, T2KORGDISK
>>>>>>> Authorized FQANs: all VOs
>>>>>>> Space Type: Any Retention Policy: Replica
>>>>>>> Number of file systems : 20 FS selection policy: maxfreespace
>>>>>>>
>>>>>>>
>>>>>>> SPACE RESERVATIONS:
>>>>>>>
>>>>>>> ATLASDATADISK ID=0cd8b32a-7561-4ebe-82e2-eabf69aea600
>>>>>>> CAPACITY: 232.00T RESERVED: 232.00T UNAVAIL (free): 0
>>>>>>> USED: 208.20T FREE: 23.80T (10.3%)
>>>>>>> .........
>>>>>>>
>>>>>>> I 'm trying to check occupied disk space by running ls -l command on file systems like
>>>>>>>
>>>>>>> ls -l /storage1/*/2013-10-12.
>>>>>>>
>>>>>>> After this check I blame biomed users.
>>>>>>>
>>>>>>> I'm wondering is there a clever way to check disk usage for last day (I don't think dpns-ls is good for this) and can i reserve disk space for ops nagios tests/ or is it possible to block biomed users for writing ?
>>>>>>>
>>>>>>>
>>>>>>> Your help is much appreciated
>>>>>>>
>>>>>>> Elena
>>>>>>>
>>>>>>> __________________________________________________
>>>>>>> Dr Elena Korolkova
>>>>>>> Email: [log in to unmask]
>>>>>>> Tel.: +44 (0)114 2223553
>>>>>>> Fax: +44 (0)114 2223555
>>>>>>> Department of Physics and Astronomy
>>>>>>> University of Sheffield
>>>>>>> Sheffield, S3 7RH, United Kingdom
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> __________________________________________________
>>>>>> Dr Elena Korolkova
>>>>>> Email: [log in to unmask]
>>>>>> Tel.: +44 (0)114 2223553
>>>>>> Fax: +44 (0)114 2223555
>>>>>> Department of Physics and Astronomy
>>>>>> University of Sheffield
>>>>>> Sheffield, S3 7RH, United Kingdom
>>>>
>>>> __________________________________________________
>>>> Dr Elena Korolkova
>>>> Email: [log in to unmask]
>>>> Tel.: +44 (0)114 2223553
>>>> Fax: +44 (0)114 2223555
>>>> Department of Physics and Astronomy
>>>> University of Sheffield
>>>> Sheffield, S3 7RH, United Kingdom
>>>>
>>>>
>>>>
>>>>
>>
>> __________________________________________________
>> Dr Elena Korolkova
>> Email: [log in to unmask]
>> Tel.: +44 (0)114 2223553
>> Fax: +44 (0)114 2223555
>> Department of Physics and Astronomy
>> University of Sheffield
>> Sheffield, S3 7RH, United Kingdom
>>
>>
>>
>>
--
John Bland [log in to unmask]
Research Fellow office: 220
High Energy Physics Division tel (int): 42911
Oliver Lodge Laboratory tel (ext): +44 (0)151 794 2911
University of Liverpool http://www.liv.ac.uk/physics/hep/
"I canna change the laws of physics, Captain!"
|