Print

Print
About QE reweighing, I have mentioned this *many* times now. 
I will try to be as clear as I can, and please try to understand: The proposed fix introduces a new and worst bug.

The QE event generation is *required* to store the actual x-section it used for generating events.

Before, it was calculating something like d2sig/dEl dccosthetal or d2sig / dq0 dq3 (probably the former) but, when it was storing that differential x-section value along with an associated kinematic phase space enumeration, it was reporting that it was dsig/dQ2. The value was correct but the enum was wrong. This was a bug.

The solution is *not* to change the actual value stored to x-section to be dsig/dQ2!! Although with this solution the value and the type are consistent, they are *both* wrong,
whereas before at least the value was correct (and things could still be salvaged).

The solution is to correct the kinematic phase space enumeration!

The reweight should read the kinematic phase space enumeration, interpret the stored value correctly, do the tweaked x-section calculation in correct kinematic space and form the weight using a ratio of differential cross-section computed in the same kinematic space (unless you want your weights to have units of GeV).

The reweight can be generic to handle reweight in d2sig/dEl dccosthetal, d2sig / dq0 dq3, dsig/dQ2, or whatever - the structure of the code is identical:

weight = tweaked_model->XSec(interaction, kinematic_phase_space) / stored_xsec_in_given_kinematic_phase_space

If we now generate events using d2sig/dEl dccosthetal or d2sig / dq0 dq3, we can not assume that all lines of constant Q2 (in El, costheta or q0,q3 spaces) will always have identical weights!
And this is why why need to store the actual x-section used for kinematical selection and not one we calculate separately. Isn’t this amply clear??

As an aside: 

I hope we are learning lessons with some of the codes that were developed without planning / designing / documenting, without thinking all aspects through before starting to type lines in the editor. They are codes that we have very hard time understanding, evaluating, improving now. Again, a detailed document is a pre-requisite for any new development - We want to see that there is thought put into the design of the code, the integration with and potential ramifications to other parts of GENIE etc.
This project *should* have been caught much earlier, during the development of the new QE generator!!

Creating an incubator project does not amount to just adding a name in the GENIE web page. It amounts to getting that document written, reviewed and agreed.
We have 22 projects (that I know of, and I should have known them all) in incubation, all of which have a GENIE docDB number (see web page) with a skeleton for a detailed planning document (scope / deliverables / milestones / requirements / outline of development plan / validation plan / review points / interdependencies with other projects / references etc etc)
This needs to be done properly (and not just mock it up by adding a couple of lines) for all projects we are working on. 

cheers
C

On 4 Dec 2018, at 03:35, Dytman, Steven A <[log in to unmask]> wrote:

Hi,
Interesting mtg this morning, always good when experts in UK get together
with people here.

- Robert's changes are now in Generator and ReWeight, look to be acceptable to all.
- We heard it's not right to make branches off master, so all branches were moved
to private forks.
- Steven G and I are nearing end of changes for FSI, will also be in both Gen and Rew.
They are all in newrew branches.  As Marco says, almost all are just substituting
new include files in all the right places.  We are now changing input files
so that unitarity errors in ReW are avoided.
- most interesting discussion was on CCQE reweighting.  I think we all agreed that there
are 2 issues:
  * the stored cross section (used for ReW) is not the same as the cross section used
for event generation.  This causes a problem in ReW, not in Gen.  It can be called a
bug or an inconsistency and can easily be fixed with a change in EventGeneratorCCQE.cxx
as Steven G. has proposed in gardiner-ccqefix branch.
  * no one at the mtg knows why we report cross section using Q2fE phase space.  None
of use know what is correct xs  to use at this time.

We *must* make a strategy for release.  We have a document describing changes nearly
completed.  LarSoft people will try *test* build of entire package tomorrow using
GENIE as of last Thur this week. 

Everything except 2nd CCQE issue is ready or close to ready.  Feel free to ask questions or
make corrections.

Steve

On 12/3/2018 7:58 AM, Steven Dytman wrote:
[log in to unmask]" class=""> Hi,

good question, we don't have a release yet so I suggest we try to meet.
Sorry, no prior notice.
Please connect at 9am CT.
Far osc room at FNAL
zoom 412-624-9244 outside FNAL.

regards,
Steve

On 12/3/2018 4:08 AM, Marco Roda wrote:
[log in to unmask]" class="">
Hi Steve,

are we having a meeting today?

Cheers,
Marco

Il giorno dom, 02/12/2018 alle 20.10 +0000, Constantinos Andreopoulos ha scritto:
Yes, it is trivial to re-release same code under different version (aka, the thing that, earlier in this thread, you thought would never have to happen). No one argued that it is not trivial. The argument is that it is not sane to do so and to have multiple releases of the exact same code. This has never been done *ever*, for any software.Just think of our communications with experiments: “Following the new Generator release, please make sure you update the ReWeight package!! Frankly, nothing changed, but it features our fancy new version number - the highest ever used in our GENIE ReWeighting product!!!”

For reasons stated many times, it will be 1.0.0, or 50.0.0 if one prefers, but it will not track the Generator (especially since we have the  expressed strategy of integrating ReWeight with Professor, in which case the ReWeight will be even more decoupled from the specifics of the Generator code, and it will hardly ever change). In addition, we can have a bug fix release (3.0.2) of a *non-existent* release (3.0.0). 
We will install checks within the ReWeight code, issue warnings and exit if we detect that a wrong version of the Generator is used, and we will maintain a table (along with all other tables that inform users about tunes etc). 

For the love of god, stop making up GENIE release strategy at the Fermilab canteen, and  let’s stop arguing about this and focus on fixing the frigging problem. The solution put forward is almost certainly wrong. QE events are now generated using a model that provides a double-differential x-section, so we can no longer reweight based on calculating variations of dsig/dQ2 alone. We are nowhere near a ReWeight release I think. Not only that, we also *have* to issue a new bug fix release of 2.12, that is still the main release used by experiments.

C

-- 
Dr. Costas Andreopoulos, FHEA
Associate Professor
University of Liverpool and STFC/RAL

Sent from my iPhone

On 2 Dec 2018, at 18:19, Dytman, Steven A <[log in to unmask]> wrote:

Steven, Robert, and I discussed strategy on Fri.  He said it's trivial to re-release the same code
under a new version label in Git.  My impression was that the old release would disappear.  He can
add his thoughts.  We all firmly agree that 3.0.2 is best for 1st ReWeight release for reasons
stated various times. 

Steve

On 11/30/2018 10:58 AM, Constantinos Andreopoulos wrote:
[log in to unmask]" type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex" class="">
What gave me the idea that we will have to cut a new ReWeight release for each new Generator release?
The clue was Robert mentioning it _explicitly_ :

[log in to unmask]" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex" class="">
[log in to unmask]" type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex" class="">
[log in to unmask]" type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex" class="">
That is, new tags of Generator would be accompanied by a new tag of Reweight to ensure that an old Reweight isn't used with a new Generator (as that generally is a wrong thing to do).  And Reweight would add a field to allow it to advance independently beyond the Generator that it relies on, while still indicating which version it is.


cheers
C


On 30 Nov 2018, at 15:43, Dytman, Steven A <[log in to unmask]> wrote:

ok, 1.0.0

Where did you get idea of need to cut new identical Reweight releases?
That was never proposed.  What was proposed was an attempt to start
releases with ReWeight and Generator together for each major release.
We are a little behind, so that means 3.0.2 now, will next be 4.0.0.

BTW, we see a few mismatches between ReWeighting and new tune
structure that will be addressed in next few months.

my ideas of democracy come from Thomas Jefferson and 40 years of
work in particle physics.  Trading insults is counterproductive.

Steve

On 11/30/2018 9:33 AM, Constantinos Andreopoulos wrote:
[log in to unmask]" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex" class=""> Absolutely does not worth the argument - It is 1.0.0. 

Generator and ReWeight are connected but _different_ products with _different_ releases.
We will not nullify the benefits of this separation by having a scheme where we always tag both of them together.

It is likely that we will _not_ have to issue another ReWeight release till we get to GENIE4 in ~Q1/2020.
Over that time, we may have ~5 or so minor Generator releases, each with as many as ~10 revisions.
It is insane to create 50 tags of the exact same ReWeight code.

Democracy? Who gave you this idea? Trump or Brexit?

C

On 30 Nov 2018, at 15:08, Dytman, Steven A <[log in to unmask]> wrote:

this is frustrating for those of us who are anxious to meet Nova and MicroBooNE deadline of Dec 1.
People I talk to have strong desire for 3.0.2, I guess you disagree and democratic principles are not applicable.
As Gabe says, problems won't be at FNAL.  I worry about other users when reweighting and generator
numbering have little relationship.

This is not worth continued argument.  What do you recommend? 1.0.0?

We expect to be done with code today.  Recent work is on branch newrew.

regards,
Steve

On 11/29/2018 01:02 PM, Constantinos Andreopoulos wrote:
[log in to unmask]" type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex" class="">
No, we did not agree on 3.0.2. In fact, I recall disagreeing strongly. See the thread below if you do not recall.

This needs to be sorted before tagging the first one. If we do not require the same version number in all products (and there was no overriding argument as to why having
 identical versions might be necessary) there is no reason to name it 3.0.2. It could be 1.0.0. 

C

-- 
Dr. Costas Andreopoulos, FHEA
Associate Professor
University of Liverpool and STFC/RAL

Sent from my iPhone

On 29 Nov 2018, at 16:39, Dytman, Steven A <[log in to unmask]> wrote:

No consensus obvious to me.  I think we are all agreed that new ReWeight release
will be 3.0.2, still need to sort out what happens after that.

Steve

On 11/28/2018 05:34 PM, Gabriel Nathan Perdue wrote:
[log in to unmask]" type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex" class=""> Also, at Fermilab we can use UPS to ensure ReWeight and Generator compatibility, no? We have two kinds of GENIE clients - experiments and model developers. We should be able to manage the complexity for experiments through tools like UPS. Model developers are a hardy bunch and can be asked to check a table. Of course, that means we should provide a table somewhere of recommended versions of ReWeight for a given version of the Generator, and vice versa.

Gabriel Perdue
Scientist

Quantum Science
Office of the CRO
Fermi National Accelerator Laboratory
PO Box 500, MS 234, Batavia, IL 60510, USA
Office: 630-840-6499
Cell: 630-605-8062

Connect with us!

On Nov 28, 2018, at 3:50 PM, Constantinos Andreopoulos <[log in to unmask]> wrote:

If we are scared that things will go terribly wrong, it is trivial to make sure it never happens!
We can guarantee that a version of ReWeight runs with a specific version of the Generator only. The Generator version is available within the ReWeight initialisation, so ReWeight can exit if it thinks it is incompatible with the version of the Generator. 

Software interdependencies are complex and matching version numbers for 2 products in particular is doing nothing much. What if our predictions start to depend critically upon the version of PYTHIA used so that our tunes and reweights do depend on a specific version of PYTHIA? Do we throw the PYTHIA version number in the mix too?

Now, move forward to the distant time that all products (Comparisons, Tuning) might also be public. Do we tag Tuning with the versions of all Generator, Reweight and Comparisons because 
it depends on all of them?

Matching version numbers is extremely monolithic and non scalable. It only works with 2 products and we already 
a) had to add a 4th number in, while
b) we reduce the independent Reweight version numbers to only 1. 
So, no matter whether you change an x to a y, or you completely rewrite the whole thing (for a given generator version), the ReWeight version is bumped up by the same amount and there is nothing in the version number to indicate the scope of changes.

Sorry, I think the proposal will eventually turn out to be unworkable for one reason or another. Like a bike with square wheels. A scheme with Independent version numbers (bumped up in a sane manner, as expected with most other codes), a table to indicate allowed or desired dependencies , and run-time checks if we are too worried, is both the simplest scheme and imposes no constraint whatsoever. 

If only our OS package manager was working out interdependencies requiring common version numbers.

cheers
C

-- 
Dr. Costas Andreopoulos, FHEA
Associate Professor
University of Liverpool and STFC/RAL

Sent from my iPhone

On 28 Nov 2018, at 18:08, Robert W Hatcher <[log in to unmask]> wrote:


On Nov 28, 2018, at 11:43 AM, Constantinos Andreopoulos <[log in to unmask]> wrote:

4 numbers? With a new number enumerating new features appended after the revision? 

New features or bug fixes to Reweight

Generator remains having only 3 fields

So basically a numbering scheme like

[(major features) - (minor features) - (bug fix vrs on previous set of features)] - (yet more features??)

The 4th seems out of place.

 No, this is the first I am hearing about this 

Well, I distinctly remember a discussion w/ Marco, and I thought you were online at the time.

I think of this scheme for Reweight as   {GeneratorVersion}_{ReweightSubVersion}.

If you don't like {GeneratorVersion}_{pkgVersion} how about {GeneratorVerson} for the first compatible Reweight, and followed by {GeneratorVersion}A, {GeneratorVersion}B ...

Whether it's adopted for Comparisons or Tuning doesn't matter to me at all.  Run those independently if one likes, because experts will know which can be used with which.   But for public-facing bits such as Reweight we must make sure that it is easy and memorable for users to get it right.   This rule for Reweight is the first three fields must match the Generator used, while for the last one generally wants the largest value available.

Tagging a new Reweight for every new Generator ensures that one is coupling any config changes with any possible Reweight changes.   But the extension allows for Reweight-only changes to move ahead.

-robert



C
-- 
Dr. Costas Andreopoulos, FHEA
Associate Professor
University of Liverpool and STFC/RAL

Sent from my iPhone

On 28 Nov 2018, at 17:37, Robert W Hatcher <[log in to unmask]> wrote:


On Nov 28, 2018, at 11:17 AM, Dytman, Steven A <[log in to unmask]> wrote:

see below

On 11/28/2018 11:03 AM, Constantinos Andreopoulos wrote:
[log in to unmask]" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex" class=""> Well, not sure ReWeight should be tagged as 3.0.2. Keeping the Generator and ReWeight version numbers the same, is not as great idea as it might look. What if we find a bug and need to produce a new ReWeight version? Do we also produce a clone Generator release with a new tag, only so that we can keep tag numbers in sync? Do not like the potential proliferation of tags, without it being warranted by code changes, only so that all products can be kept in sync. What if we consider Comparisons and Tuning too? Do we keep everything in sync, tagging all 4 products together? Or do we introduce variations to the numbering scheme (3.0.2a etc)? Isn’t 3 numbers just enough info for organising releases? What is wrong with a simple table associating supported combinations of products.

for now, why not use same number for ReWeight and Generator?  Tuning and Comparisons are 3.0.0
so we are close together.  As you say, divergences will develop over time.  I don't think we want to
synch all 4 products.  If we have new major releases about once a year, we will usually be close.  I'd
like to hear more opinions.

Hmm.  I thought we discussed this before and agreed on the idea:

Generator Reweight
R-3_00_00 -
R-3_00_02 R-3_00_02 (or R-3_00_02_00?)
R-3_00_02_02 # new features of reweighting 
R-3_00_02_04 # yet more
R-3_00_04 R-3_00_04 (or R-3_00_04_00)
...

That is, new tags of Generator would be accompanied by a new tag of Reweight to ensure that an old Reweight isn't used with a new Generator (as that generally is a wrong thing to do).  And Reweight would add a field to allow it to advance independently beyond the Generator that it relies on, while still indicating which version it is.

I really, really don't think we want them to run independently and rely on people consulting some lookup table ... that sounds like asking for trouble.

-robert

[log in to unmask]" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex" class="">
The goal with the document is that it precedes the release. Not knowing whether more commits will follow today does not sound like a robust starting point for deciding that we have a release tomorrow? Are there residual problems or not?

I'm being careful.  We think we have reweighting working.  Steven made a script to test all
reweighting tweaks in all tunes and that shows no errors.  Robert is renaming splines to match
new names today.  I found a potential error yesterday that we are still investigating.  That is
as much as I know.

regards,
Steve

[log in to unmask]" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex" class="">
cheers
C

-- 
Dr. Costas Andreopoulos, FHEA
Associate Professor
University of Liverpool and STFC/RAL

Sent from my iPhone

On 28 Nov 2018, at 16:40, Dytman, Steven A <[log in to unmask]> wrote:

Generator release will be 3.0.2, not sure about ReWeight release.
We should keep numbering similar, how about also using 3.0.2?

Steve

On 11/28/2018 10:26 AM, Steven Dytman wrote:
Robert, Steven G, and I have been making small changes to Generator
and ReWeight.
Some commits have already been made, perhaps more today.  Our goal is
to have
new release tomorrow.

We will put together a small document and post it to docdb.

regards,
Steve

On 11/26/2018 10:27 AM, Costas Andreopoulos - UKRI STFC wrote:
Same as always? Guess we are talking strictly about a bug-fix
revision of the Generator (3.0.2) or the first tag of ReWeight
(effectively, a bug-fix revision of what we would have already
released in October had it still been part of the Generator).
So, some documentation of the bug fixes (eg a DocDB document, or info
submitted together with a pull request - whatever is most appropriate
given the scale and scope of changes) + some validation showing the
bug was fixed and nothing else was screwed by mistake. Then we can
take it from there.
C

On 26 Nov 2018, at 16:03, Dytman, Steven A <[log in to unmask]> wrote:

ok, we have important deadline very soon.  What is procedure for
agreeing
on a release?

Steve

On 11/26/2018 10:40 AM, Costas Andreopoulos - UKRI STFC wrote:
I can not connect on Wedns. I am teaching and then I am travelling
to IFIC.
C

On 26 Nov 2018, at 14:12, Dytman, Steven A
<[log in to unmask]<mailto:[log in to unmask]>> wrote:

excellent idea, is that all right with you, Costas?

On 11/26/2018 7:54 AM, Marco Roda wrote:
Hi Steve,

   ok, no problem.
Maybe we could use the time slot we had this summer for the tuning: it
was Wednesday at 10 am Chicago time.

Cheers,
    Marco

Il giorno lun, 26/11/2018 alle 13.41 +0000, Dytman, Steven A ha
scritto:
There is a significant snow storm in the Chicago area, not sure what
to do.
Researchers here coming back from elsewhere surely have lots of
trouble
yesterday and today.  Those of us here have up to 1 ft=30 cm of snow.

Meeting is postponed, but we still have to decide when we are done
with
changes to reweighting.  I'm thinking midweek, we'll have to stay in
touch
via email & slack.  Steven and I are still doing tests and Robert
was
supposed
to have info on splines early this week.

Steve

#####################################################################
###

To unsubscribe from the NEUTRINO-MC-CORE list, click the following
link:
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jiscmail.ac.uk%2Fcgi-bin%2Fwebadmin%3FSUBED1%3DNEUTRINO-MC-CORE%26A%3D1&amp;data=02%7C01%7Cdytman%40pitt.edu%7C892f14d4573c4dcd1c5c08d653bc2048%7C9ef9f489e0a04eeb87cc3a526112fd0d%7C1%7C0%7C636788464802973660&amp;sdata=gCZNprjdR5tPIivPrWxqr%2F%2FydwuTaMBbyubd7zreA4I%3D&amp;reserved=0



########################################################################


To unsubscribe from the NEUTRINO-MC-CORE list, click the following
link:
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jiscmail.ac.uk%2Fcgi-bin%2Fwebadmin%3FSUBED1%3DNEUTRINO-MC-CORE%26A%3D1&amp;data=02%7C01%7Cdytman%40pitt.edu%7C892f14d4573c4dcd1c5c08d653bc2048%7C9ef9f489e0a04eeb87cc3a526112fd0d%7C1%7C0%7C636788464802973660&amp;sdata=gCZNprjdR5tPIivPrWxqr%2F%2FydwuTaMBbyubd7zreA4I%3D&amp;reserved=0





########################################################################

To unsubscribe from the NEUTRINO-MC-CORE list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=NEUTRINO-MC-CORE&A=1


To unsubscribe from the NEUTRINO-MC-CORE list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=NEUTRINO-MC-CORE&A=1



To unsubscribe from the NEUTRINO-MC-CORE list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=NEUTRINO-MC-CORE&A=1



To unsubscribe from the NEUTRINO-MC-CORE list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=NEUTRINO-MC-CORE&A=1



To unsubscribe from the NEUTRINO-MC-CORE list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=NEUTRINO-MC-CORE&A=1



To unsubscribe from the NEUTRINO-MC-CORE list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=NEUTRINO-MC-CORE&A=1



To unsubscribe from the NEUTRINO-MC-CORE list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=NEUTRINO-MC-CORE&A=1




To unsubscribe from the NEUTRINO-MC-CORE list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=NEUTRINO-MC-CORE&A=1




To unsubscribe from the NEUTRINO-MC-CORE list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=NEUTRINO-MC-CORE&A=1 

--
Marco Roda, PhD in Physics
Post Doctorate Research Associate

University of Liverpool
Department of Physics 
Oliver Lodge Laboratory
Liverpool L69 7ZE, UK

Office: +44 (0)151 79 43403 
Mobile: +44 (0)745 381 2081

To unsubscribe from the NEUTRINO-MC-CORE list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=NEUTRINO-MC-CORE&A=1




To unsubscribe from the NEUTRINO-MC-CORE list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=NEUTRINO-MC-CORE&A=1



To unsubscribe from the NEUTRINO-MC-CORE list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=NEUTRINO-MC-CORE&A=1