[Suggestion] 1.12.2 Config option for list of sounds volume and disabling

One of the other sound mods I use, Ambient Sounds 3, has a config file that allows the volume of each individual sound to be changed or muted. Would like to see the same or similar options for Dynamic Surroundings.
I may want to use this mod for certain sounds like crafting, armor, etc., but have other mods for other environmental sounds. Or people may just want to disable bird screeching or specific animal sounds in certain biomes.
This would be a nice feature to see in a future version.
Thanks so much and keep up the great work.

In line with this suggestion, I have had people on my server complaining about the snake noises in deserts and the elephant sounds in savannas when there are no such animals around ( and we do have those animals added by other mods, so they could be), but it seems the only option is to turn odd the entire biome-specific sound system, which is sad.

Strange thing is, that even with this turned off:
# Enable biome background and spot sounds [default: true]
B:"Enable Biomes Sounds"=false
You still get the desert and savanna animal sounds.

I'm a bit confused.  Dynamic Surrounds has a rather large configuration section for tuning individual sounds.  Using the mod options GUI navigate to Dynamic Surroundings.  The first button at the top should be "Individual Sound Configurations".  Let me know if you are not seeing it.
As for the animal sounds those are ambient sounds, not biome.  Biome sounds are the sounds that play like a sound track (wind in mounts, general bird chirps and leaf rustle in forest, crickets at night in plains, etc.)
As for snakes and elephants, yes, those sounds are in the mod.  If they don't want those sounds they can turn off in "Individual Sound Configurations".

Only ever tried to adjust settings in the config.  Didn't try the ingame config, as I figured it would have the same options.  We ended up taking this off, because our players hated the snake sounds.  Will do a test on it at some point and look at the ingame config.

FYI - the next update (<IP_ADDRESS>) will have some of the sounds players find annoying disabled as a default.  Snake is one.  I will mark this issue as fix because of that.

Pushed v<IP_ADDRESS> to CurseForge.


InterfaceError: Error binding parameter 1 - probably unsupported type.

Describe the bug
When running pytest with the pytest-cov package (includes coveragepy), an INTERNALERROR is thrown. This looks similar to #1010 @nedbat , but it seems like the bug is still not fixed as my project is using v5.5.0 of coverage.
To Reproduce
Run command pytest --cov-report=term-missing --ignore=./test/functional/ --cov=onionbalance
NB. if the above command is ran without the cov option, no internal error is thrown, BUT the coverage report is not generated still
platform darwin -- Python 3.9.1, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
rootdir: /Users/***/Desktop/onionbalance, configfile: setup.cfg
plugins: cov-2.11.1, mock-3.5.1
collected 46 items

test/v2/test_consensus.py ....                                            [  8%]
test/v2/test_descriptor.py .....                                          [ 19%]
test/v2/test_settings.py ..                                               [ 23%]
test/v2/test_util.py ............................                         [ 84%]
test/v3/test_v3_descriptor.py .                                           [ 86%]
test/v3/test_v3_hashring.py .                                             [ 89%]
test/v3/test_v3_keys.py .                                                 [ 91%]
test/v3/test_v3_onionbalance.py ..                                        [ 95%]
test/v3/test_v3_status.py ..                                              [100%]
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/_pytest/main.py", line 269, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/_pytest/main.py", line 323, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/pluggy/hooks.py", line 286, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/pluggy/manager.py", line 93, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/pluggy/manager.py", line 84, in <lambda>
INTERNALERROR>     self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/pluggy/callers.py", line 203, in _multicall
INTERNALERROR>     gen.send(outcome)
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/pytest_cov/plugin.py", line 271, in pytest_runtestloop
INTERNALERROR>     self.cov_controller.finish()
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/pytest_cov/engine.py", line 44, in ensure_topdir_wrapper
INTERNALERROR>     return meth(self, *args, **kwargs)
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/pytest_cov/engine.py", line 230, in finish
INTERNALERROR>     self.cov.stop()
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/coverage/control.py", line 659, in save
INTERNALERROR>     data = self.get_data()
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/coverage/control.py", line 727, in get_data
INTERNALERROR>     if self._collector and self._collector.flush_data():
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/coverage/collector.py", line 442, in flush_data
INTERNALERROR>     self.covdata.add_lines(self.mapped_file_dict(self.data))
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/coverage/sqldata.py", line 438, in add_lines
INTERNALERROR>     self._choose_lines_or_arcs(lines=True)
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/coverage/sqldata.py", line 495, in _choose_lines_or_arcs
INTERNALERROR>     with self._connect() as con:
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/coverage/sqldata.py", line 300, in _connect
INTERNALERROR>     self._create_db()
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/coverage/sqldata.py", line 252, in _create_db
INTERNALERROR>     db.executemany(
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/coverage/sqldata.py", line 1106, in executemany
INTERNALERROR>     return self.con.executemany(sql, data)
INTERNALERROR> sqlite3.InterfaceError: Error binding parameter 1 - probably unsupported type.

Expected behavior
I expected a coverage report to be generated without throwing an InternalError.

Thanks for the report.  Is this reproducible for you?  It looks like the "fix" for #1010 needs to be applied in another place, though you are not using PyPy, which was a key element in #1010.
Can you try this patch?
diff --git a/coverage/sqldata.py b/coverage/sqldata.py
index a150fdfd..f58486ac 100644
--- a/coverage/sqldata.py
+++ b/coverage/sqldata.py
@@ -1103,7 +1103,14 @@ def executemany(self, sql, data):
         if self.debug:
             data = list(data)
             self.debug.write("Executing many {!r} with {} rows".format(sql, len(data)))
-        return self.con.executemany(sql, data)
+        try:
+            return self.con.executemany(sql, data)
+        except Exception:
+            # In some cases, an error might happen that isn't really an
+            # error.  Try again immediately.
+            # https://github.com/nedbat/coveragepy/issues/1010
+            # https://github.com/nedbat/coveragepy/issues/1138
+            return self.con.executemany(sql, data)

     def executescript(self, script):
         """Same as :meth:`python:sqlite3.Connection.executescript`."""

@nedbat Thanks for the quick response. However, I still get an INTERNALERROR. What I did? I tried patching it by first pip uninstalling coverage and then installed coverage from your branch ( 5.5.1a0 ).
Have you got any other insights as to how this can be fixed?
See the logs:
pytest --cov-report=term-missing --ignore=./test/functional/ --cov=onionbalance
============================== test session starts ==============================
platform darwin -- Python 3.9.1, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
rootdir: /Users/***/Desktop/onionbalance, configfile: setup.cfg
plugins: cov-2.11.1, mock-3.5.1
collected 46 items

test/v2/test_consensus.py ....                                            [  8%]
test/v2/test_descriptor.py .....                                          [ 19%]
test/v2/test_settings.py ..                                               [ 23%]
test/v2/test_util.py ............................                         [ 84%]
test/v3/test_v3_descriptor.py .                                           [ 86%]
test/v3/test_v3_hashring.py .                                             [ 89%]
test/v3/test_v3_keys.py .                                                 [ 91%]
test/v3/test_v3_onionbalance.py ..                                        [ 95%]
test/v3/test_v3_status.py ..                                              [100%]
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/coverage/sqldata.py", line 1107, in executemany
INTERNALERROR>     return self.con.executemany(sql, data)
INTERNALERROR> sqlite3.InterfaceError: Error binding parameter 1 - probably unsupported type.
INTERNALERROR>
INTERNALERROR> During handling of the above exception, another exception occurred:
INTERNALERROR>
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/_pytest/main.py", line 269, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/_pytest/main.py", line 323, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/pluggy/hooks.py", line 286, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/pluggy/manager.py", line 93, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/pluggy/manager.py", line 84, in <lambda>
INTERNALERROR>     self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/pluggy/callers.py", line 203, in _multicall
INTERNALERROR>     gen.send(outcome)
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/pytest_cov/plugin.py", line 271, in pytest_runtestloop
INTERNALERROR>     self.cov_controller.finish()
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/pytest_cov/engine.py", line 44, in ensure_topdir_wrapper
INTERNALERROR>     return meth(self, *args, **kwargs)
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/pytest_cov/engine.py", line 230, in finish
INTERNALERROR>     self.cov.stop()
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/coverage/control.py", line 659, in save
INTERNALERROR>     data = self.get_data()
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/coverage/control.py", line 727, in get_data
INTERNALERROR>     if self._collector and self._collector.flush_data():
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/coverage/collector.py", line 442, in flush_data
INTERNALERROR>     self.covdata.add_lines(self.mapped_file_dict(self.data))
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/coverage/sqldata.py", line 438, in add_lines
INTERNALERROR>     self._choose_lines_or_arcs(lines=True)
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/coverage/sqldata.py", line 495, in _choose_lines_or_arcs
INTERNALERROR>     with self._connect() as con:
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/coverage/sqldata.py", line 300, in _connect
INTERNALERROR>     self._create_db()
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/coverage/sqldata.py", line 252, in _create_db
INTERNALERROR>     db.executemany(
INTERNALERROR>   File "/usr/local/lib/python3.9/site-packages/coverage/sqldata.py", line 1113, in executemany
INTERNALERROR>     return self.con.executemany(sql, data)
INTERNALERROR> sqlite3.IntegrityError: UNIQUE constraint failed: meta.key

It seems like you have an unusual situation. Can you give me a way to reproduce it?

Though: the error you report here could just be that it's too simplistic to retry all of the statements in an executemany, since some will have succeeded by the time one of them fails, and retrying all of them will cause the constraint failure we're seeing.
Either way, it would be helpful to have a way to reproduce it.

@nedbat Yes, absolutely. To reproduce, run the following commands for the this dev branch https://github.com/felipejinli/onionbalance/tree/dev.
0. should be on root project directory onionbalance

pip install -r requirements.txt
pip install -r test-requirements.txt
pytest --cov-report=term-missing --ignore=./test/functional/ --cov=onionbalance
pip uninstall coverage
pip install git+https://github.com/nedbat/coveragepy.git@nedbat/bug1138#egg=coverage
PS. I'm unsure as to how I should link your branch's coverage patch into requirements.txt (say in a situation where the patch fully fixed it since). Have you got any suggestions?
Regarding reproducibility, lmk if you have any issues. Thanks!

Thanks for the reproducible case.
The problem is your mock of datetime.datetime in test_outdated_consensus.  You never undo that mock, so datetime.datetime is a mock object for the rest of the test run.  Coverage.py later tries to get the current time to put in its database, and gets a mock object instead of a timestamp.  SQLite can't write the mock object to the database.
The best thing to do would be to use mock.patch as a context manager instead.  It will automatically clean up the mock at the end of the test:
    def test_outdated_consensus(self):
        current_time = datetime.datetime.fromtimestamp(10101010101)

        consensus = DummyConsensus()

        consensus.consensus = mock.Mock()
        with mock.patch("datetime.datetime") as mock_datetime:
            consensus.consensus.valid_after = current_time
            # valid_until is 3 hours in the future
            consensus.consensus.valid_until = current_time + datetime.timedelta(seconds=3600*3)

            # Test some legitimate cases
            mock_datetime.utcnow.return_value = current_time
            self.assertTrue(consensus.is_live())

            mock_datetime.utcnow.return_value = current_time + datetime.timedelta(seconds=3600*11)
            self.assertTrue(consensus.is_live())

            mock_datetime.utcnow.return_value = current_time + datetime.timedelta(seconds=3600*12)
            self.assertTrue(consensus.is_live())

            mock_datetime.utcnow.return_value = current_time + datetime.timedelta(seconds=3600*24)
            self.assertTrue(consensus.is_live())

            mock_datetime.utcnow.return_value = current_time - datetime.timedelta(seconds=3600*24)
            self.assertTrue(consensus.is_live())

            # Now test some bad cases. The is_live() function is lenient up to 24
            # hours after the valid_until, or 24 hours before the valid_after
            mock_datetime.utcnow.return_value = consensus.consensus.valid_until + datetime.timedelta(seconds=3600*24+1)
            self.assertFalse(consensus.is_live())

            mock_datetime.utcnow.return_value = consensus.consensus.valid_after - datetime.timedelta(seconds=3600*24+1)
            self.assertFalse(consensus.is_live())

As another tip: it's better to mock where you use an object, rather than where it is defined.  That will limit the scope to just the code that needs the mock.  That might be harder in this case, it looks like a few different places use utcnow.


feature: Kangaroo12

Implement support for parallel hashing. At a minimum, the specification itself should be implemented according to Keccak team specs. With rust, it's possible the highly optimized LLVM derived compiler may automatically run the hash as SIMD.
Alternatively, we could implement NIST ParallelHash with rayon. Measuring the performance of the library as the input scales should be valuable.

Optimized C implementation:
Python reference implementation
Rust xkcp implementation

closing this because sha3 is already heavily parallelized at the machine level and k12 is not really used in practice


Frontend Updates

Some changes and updates are required.

Signup

[ ] Remove the Company and the Team size fields

Login

[ ] Some spaces between elements

Account Management

[ ] Transfer the password field to another tab
[ ]  Add Delete Account
[ ] No Saved card mockup

https://www.figma.com/proto/5wo9b6Sik1J0z4eLf0CHzB/cloud4students?node-id=676-321&t=qdBzcrMRJk937tDm-0&scaling=scale-down&content-scaling=fixed&page-id=0%3A1
all updated

[ ] made more space at sign-in to align with sign-up
[ ] remove all icons from all inputs to be consistent
[ ] typo Create a New Virtual Machine( add a)
[ ] Remove (*) from Name and Region at Create VM page and maybe add a message below the field
"Machine Name and Region are required to deploy the VM. Please fill in both fields"
[ ] update payment with Voucher (Cobone)
[ ] Make Choose your TOP-up amount auto-updated when the user chooses 50 or 100
[ ] Remove password from profile(account) and add it at new tap (change password)
[ ] update Account to be Profile at Account management
[ ] Add Delete Account to Account Management.

@ehab-hassan As per @rawdaGastan discussion with the current endpoints we need in

Deploy page:

[ ] checkbox on every VM card for public IP so the user can check whether he needs it.

Work in Progress (WIP):

Region field updated with the data
Account management updated except payments tab

Work in Progress (WIP):

Implement Delete Account tab
Add list Invoices

working on stripe implementation.

Work in Progress (WIP):
cannot import stripe , it returns undefined

Work in Progress (WIP):
Stripe imported successfully, working on implementing functions

Work in Progress (WIP):

implement .env vars
Handle get cards and add cards

Work Completed:

Voucher part implemented
Topup modification is done


Syntax fix in tutorial-ingress-controller-add-on-new.md

Proposed change: Syntax fix

Thanks, @joeyc-dev!

Due to the upcoming Build conference, all merging and publishing is on hold until the morning of 5/21.
This PR is ready for merge on 5/21.
#label: "aq-pr-triaged"
#label: "event-triaged-to-10AM"
#label: "missed-content-complete"


Results Page Card Layout and Buttons

Issuer: Kristin
Assignees: Kristin, Maryam
Requirements:

[x] try to prevent scrolling (2 different ways: have all card images lined up with descriptions below them or have 2 cards with their descriptions on one row)
[x] add a button to the home page matching the button design from index.css
[ ] try to match the next and previous button with the button style from index.css as well to see if it looks better

Deadline:
Thursday 6/8
Code-dependency:
results.js
Issue done criteria:

[x] requirements above are met

Merged with PR #81


Managing existing virtmanager vms

Hi, I would like to manage my existing vms, Linux and Windows, with QuicGui. Is that possible?
Thanks, Ray

Hello,
quickgui is a wrapper around quickget and quickemu. It can only manage virtual machines created by those tools.
If, one day, quickemu supports running virtual machines created by other tools. quickgui will also support that feature.
But for now, quickgui cannot do what you ask. Sorry.


Unable to Create Conda Environment

I wanted to follow along with one of the tutorials, but after cloning and repository and going into the biobb_wf_virtual-screening folder, I see the following after trying to create the conda environment:
Channels:

conda-forge
bioconda
defaults
Platform: win-64
Collecting package metadata (repodata.json): done
Solving environment: failed
Channels:
conda-forge
bioconda
defaults
Platform: win-64
Collecting package metadata (repodata.json): done
Solving environment: failed

LibMambaUnsatisfiableError: Encountered problems while solving:

nothing provides ambertools >=22.0 needed by biobb_chemistry-4.1.0-pyhdfd78af_0
nothing provides fpocket >=<IP_ADDRESS> needed by biobb_vs-4.1.1-pyhdfd78af_0

Could not solve for environment specs
The following packages are incompatible
├─ biobb_chemistry 4.1.0  is not installable because it requires
│  └─ ambertools >=22.0 , which does not exist (perhaps a missing channel);
└─ biobb_vs 4.1.1  is not installable because it requires
└─ fpocket >=<IP_ADDRESS> , which does not exist (perhaps a missing channel).
I just completed the 2 first steps of the conda installation process and then encountered the above error when I tried the third step.
I was expecting the conda environment to be created so then I could activate it and follow along with one of the tutorials.

OS: Windows 11
Python version: 3.10
BioBB version ??

Hi, both of these packages, fpocket and ambertools, are not compatible with windows. So, as the two of them are dependencies of biobb_vs and biobb_chemistry, you get this error.
Try in ubuntu or mac. If you don't have none of these operating systems, then enable the WSL in your Windows:
https://mmb.irbbarcelona.org/biobb/availability/tutorials/windows
Best.


Add summary about checks across platforms to /pkgs

To make it quick to see if there's >0 or 0 errors/warnings/notes

{
  "summary": {
     "ok": 0,
     "note": 0,
     "warning": 0,
     "error": 0
  }
}

working on this, just need to run on server soon.


Actual parameter setting for different sizes of the model?

 parser.add_argument(
        "--total-token",
        type=int,
        default=60,
        help="The maximum number of new generated tokens.",
    )
    parser.add_argument(
        "--depth",
        type=int,
        default=5,
        help="The maximum number of new generated tokens.",
    )

I noticed the actual depth is args.depth+1, and actual total-token is args.total_token - 1.
refer to this part of the code:
class Model(nn.Module):
    def __init__(self, config, load_emb=False, path=None, bias=True, total_tokens=63, depth=5, top_k=8, threshold=1.0):
        super().__init__()

        self.gradient_checkpointing = True
        self.padding_idx = config.pad_token_id
        self.vocab_size = config.vocab_size

        self.embed_tokens = nn.Embedding(config.vocab_size, config.hidden_size, self.padding_idx)
        if load_emb:
            from safetensors import safe_open
            import json
            try:
                with open(os.path.join(path, "model.safetensors.index.json"), "r") as f:
                    index_json = json.loads(f.read())
                    emb_path = index_json["weight_map"]["model.embed_tokens.weight"]
                with safe_open(os.path.join(path, emb_path),
                               framework="pt",
                               device="cpu") as f:
                    tensor_slice = f.get_slice("model.embed_tokens.weight")
                    vocab_size, hidden_dim = tensor_slice.get_shape()
                    tensor = tensor_slice[:, :hidden_dim].float()
            except:
                with open(os.path.join(path, "pytorch_model.bin.index.json"), "r") as f:
                    index_json = json.loads(f.read())
                    emb_path = index_json["weight_map"]["model.embed_tokens.weight"]
                weights = torch.load(os.path.join(path, emb_path))
                tensor = weights["model.embed_tokens.weight"].float()
            self.embed_tokens.weight.data = tensor

        self.top_k = top_k
        self.total_tokens = total_tokens - 1
        self.depth = depth
        self.threshold = math.log(threshold)

Could you please show me your settings for different sizes of the model?

The actual setting is as mentioned in the paper. The total-token in the code does not include the one token generated by the target model, so it is reduced by one.


Update pioneers-exercise-2.md

change build-enterprise to just build

Has been fixed by a different PR.


Duplicate lumisection in single workflow

For this workflow which is the only one writing in its output
pdmvserv_TOP-RunIISummer15wmLHEGS-00054_00206_v0__161102_162051_8417
event completion real 1999031 expected 2000000
event completion real 1997739 expected 2000000
pdmvserv_TOP-RunIISummer15wmLHEGS-00054_00206_v0__161102_162051_8417 has duplicates
{
"/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/RunIISummer15wmLHEGS-MCRUN2_71_V1-v1/GEN-SIM": false,
"/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/RunIISummer15wmLHEGS-MCRUN2_71_V1-v1/LHE": true
}
{
"/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/RunIISummer15wmLHEGS-MCRUN2_71_V1-v1/GEN-SIM": {},
"/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/RunIISummer15wmLHEGS-MCRUN2_71_V1-v1/LHE": {
"0:7592": [
"/store/mc/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/F08187C8-35A2-E611-ADBF-A0000420FE80.root",
"/store/mc/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/D09C66C9-AFA2-E611-9A52-0CC47AA98A0E.root"
],
"0:7591": [
"/store/mc/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/F08187C8-35A2-E611-ADBF-A0000420FE80.root",
"/store/mc/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/D09C66C9-AFA2-E611-9A52-0CC47AA98A0E.root"
]
}
it's the second instance in not too long so there must be something wrong
@prozober

The only thing I found in the agent (vocms0116) were logs like these [1]. @amaltaro do you think these errors could cause duplicates?
[1]
install/wmagent/JobAccountant/ComponentLog:2016-11-03 20:34:08,337:140528953976576:ERROR:AccountantWorker:Bad FwkJobReport Path: /data/srv/wmagent/v1.0.21.patch2/install/wmagent/JobCreator/JobCache/pdmvserv_TOP-RunIISummer15wmLHEGS-00054_00206_v0__161102_162051_8417/Production/JobCollection_11079_0/job_119483/Report.0.pkl
install/wmagent/ErrorHandler/ComponentLog:2016-11-03 20:36:36,076:140194997511936:ERROR:ErrorHandlerPoller:Failed to find FWJR for job 119483 in location /data/srv/wmagent/v1.0.21.patch2/install/wmagent/JobCreator/JobCache/pdmvserv_TOP-RunIISummer15wmLHEGS-00054_00206_v0__161102_162051_8417/Production/JobCollection_11079_0/job_119483/Report.0.pkl

Please keep this workflow active if possible. I'll try to have a look at the job configurations to see if they were really meant to produce the same lumi sections (bear in mind this week is not very productive in terms of real work).
Just in case, these 2 different files really produced the same lumi section 7591 (as cryptically reported above):
https://cmsweb.cern.ch/dbs/prod/global/DBSReader/filelumis?logical_file_name=/store/mc/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/F08187C8-35A2-E611-ADBF-A0000420FE80.root
https://cmsweb.cern.ch/dbs/prod/global/DBSReader/filelumis?logical_file_name=/store/mc/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/D09C66C9-AFA2-E611-9A52-0CC47AA98A0E.root

7592 is also duplicated

We won't be able to keep the workflow in that status forever, as we have to deliver it.
Any chance you can grab the information you need from the db to understand this before we deliver this back to analysis ?

Hi Jean-Roch, you can process I copied problem job reports here
/afs/cern.ch/user/c/cmst1/test/duplicatelumi
Problem seems to be there is difference in fwjr and wmbs record. (vocms0116)
Looking at the wmbs,  following is unmerged file containing (7591, 7592) lumis.
id=378511
/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/BEA7849C-FDA1-E611-B937-0CC47AD98B90.root
This file contains the above unmerged file.  id=565303 (merged one) created by job 280836
/store/mc/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/D09C66C9-AFA2-E611-9A52-0CC47AA98A0E.root
But first one does not.  (in webs_file_parent table) id=395554 created by job 266543
/store/mc/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/F08187C8-35A2-E611-ADBF-A0000420FE80.root'
But both of merged files  contains duplicate lumis although second job doesn't have the input file which contain the lumis.
select * from wmbs_file_runlumi_map where fileid=395554 and lumi in (7591, 7592);
select * from wmbs_file_runlumi_map where fileid=565303 and lumi in (7591, 7592);
Looking at the fwjr for job 266543, following input file seems to have lumis  (7591, 7592)
id=284841
/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/A685DDD8-5CA1-E611-9020-A0000420FE80.root
But wmbs has different record, (lump 7499, 7500)
select run, lumi from wmbs_file_runlumi_map wm inner join wmbs_file_details wd
on wm.fileid=wd.id where wd.lfn='/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/A685DDD8-5CA1-E611-9020-A0000420FE80.root';
1	7499
1	7500
Looking at the source code.  This is where input file information is added
https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/FwkJobReport/XMLParser.py#L82
https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/FwkJobReport/Report.py#L493
I am not completely sure how CMSSW gets the run lumi information.
But it looks like there there is difference between CMSSW framework report run lumi information and wmbs run lumi information.  There might some edge case masking problem since lumi numbers are only off by 2.

@prozober please do.
@ticoann that could be worth passing on the edm people over HN @Dr15Jones

Do you have logs and configuration files for the jobs which produced output with the same lumis?

Hi Chris, Here is the config file.
https://cmsweb.cern.ch/couchdb/reqmgr_config_cache/0e64d45e2cc39207afb73e7fd4bb62f4/configFile
Right now I only have a condor logs, @amaltaro, Alan, does logArchive job contains CMSSW log?
Anyway, logArchive contains cmssw log, here is pfns - (just for tracking purpose). (It might be already deleted)
srm://srm.rcac.purdue.edu:8443/srm/v2/server?SFN=/mnt/hadoop//store/unmerged/logs/prod/2016/11/4/pdmvserv_TOP-RunIISummer15wmLHEGS-00054_00206_v0__161102_162051_8417/Production/ProductionMergeLHEoutput/0000/0/a7ca0da2-a235-11e6-82a7-02163e017c3c-0-0-logArchive.tar.gz
srm://cmssrm.hep.wisc.edu:8443/srm/v2/server?SFN=/hdfs/store/unmerged/logs/prod/2016/11/4/pdmvserv_TOP-RunIISummer15wmLHEGS-00054_00206_v0__161102_162051_8417/Production/ProductionMergeLHEoutput/0000/0/4aadd12e-a2af-11e6-82a7-02163e017c3c-2-0-logArchive.tar.gz
If they are deleted, we can dig into logArchive tar ball containing above logs.
Let me know if you want me to send the log files via email.
Thanks,

Yes, and it should also contain the original FJR (and the PSet)

On Nov 10, 2016, at 9:42 PM, ticoann<EMAIL_ADDRESS>wrote:
Let me know if you want me to send the log files via email.

If you can make the log files, FJR and the PSet available (can just be in AFS or on the web) that would be very helpful.
Chris

Christopher Jones
Fermi National Accelerator Laboratory
<EMAIL_ADDRESS>

Of course the files are already deleted. :-(.
I think these are log files containing above logArchive files.
I am trying to copy the log I am not sure how big these are whether I have enough space. Alan do you usually copy in your home dir? (I just need to use xrdcp command, right? do you know the command I can check the size of file?)
Thanks,
root://castorcms.cern.ch//castor/cern.ch/cms/store/logs/prod/2016/11/WMAgent/pdmvserv_TOP-RunIISummer15wmLHEGS-00054_00206_v0__161102_162051_8417/pdmvserv_TOP-RunIISummer15wmLHEGS-00054_00206_v0__161102_162051_8417-ProductionLHEoutputMergeLogCollect-g31n41-3-logs.tar

root://castorcms.cern.ch//castor/cern.ch/cms/store/logs/prod/2016/11/WMAgent/pdmvserv_TOP-RunIISummer15wmLHEGS-00054_00206_v0__161102_162051_8417/pdmvserv_TOP-RunIISummer15wmLHEGS-00054_00206_v0__161102_162051_8417-ProductionLHEoutputMergeLogCollect-cms-h003-11-logs.tar

Seangchan, these are the Merge logcollects and hopefully there is only one tarball inside each of them. I copied their logArchiver over here:
http://amaltaro.web.cern.ch/amaltaro/forWMCore/a7ca0da2-a235-11e6-82a7-02163e017c3c-0-0-logArchive.tar.gz
http://amaltaro.web.cern.ch/amaltaro/forWMCore/a7ca0da2-a235-11e6-82a7-02163e017c3c-0-0-logArchive.tar.gz
where you can find information for the 2 LFNs you previously mentioned:
/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/A685DDD8-5CA1-E611-9020-A0000420FE80.root
/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/BEA7849C-FDA1-E611-B937-0CC47AD98B90.root

According to the Merge FJR, indeed two files had lumi sections 7591 and 7592.
I asked Valentin/Yuyi to search for the logArchive where those 2 files were created.

And these are the logs that produced the unmerged files:
http://amaltaro.web.cern.ch/amaltaro/forWMCore/Issue_7371/1e19574a-a152-11e6-a6e4-02163e017c3c-749-0-logArchive.tar.gz
http://amaltaro.web.cern.ch/amaltaro/forWMCore/Issue_7371/1e19574a-a152-11e6-a6e4-02163e017c3c-795-1-logArchive.tar.gz
It can be seen in these FJRs, but indeed one job produced a file with lumis 7499 and 7500 and the other job produced a file with lumis 7591 and 7592.
Seangchan, WMCore does not provide run/lumi mask for merge jobs, does it? We just provide the unmerged files to be merged and the framework takes care of it AND creates the FJR, that is then used by the agent for publishing this data and so on. Am I right?
IF so, then what needs to be figured out is why CMSSW wrote the wrong lumi sections into the FJR.

Thanks Alan for the logAchive files.

Seangchan, WMCore does not provide run/lumi mask for merge jobs, does it? We just provide the unmerged files to be merged and the framework takes care of it AND creates the FJR, that is then used by the agent for publishing this data and so on. Am I right?

Yes you are right. We don't provide run/lumi mask. @Dr15Jones, how CMSSW gets the run lumi information from unmarked file. (Is the file contains that information?)
Here are the log files Alan posted.
merge job
http://amaltaro.web.cern.ch/amaltaro/forWMCore/Issue_7371/a7ca0da2-a235-11e6-82a7-02163e017c3c-0-0-logArchive.tar.gz
http://amaltaro.web.cern.ch/amaltaro/forWMCore/Issue_7371/4aadd12e-a2af-11e6-82a7-02163e017c3c-2-0-logArchive.tar.gz
production job
http://amaltaro.web.cern.ch/amaltaro/forWMCore/Issue_7371/1e19574a-a152-11e6-a6e4-02163e017c3c-749-0-logArchive.tar.gz
http://amaltaro.web.cern.ch/amaltaro/forWMCore/Issue_7371/1e19574a-a152-11e6-a6e4-02163e017c3c-795-1-logArchive.tar.gz

Hi everyone,

On Nov 11, 2016, at 5:27 PM, ticoann<EMAIL_ADDRESS>wrote:
Thanks Alan for the logAchive files.
Seangchan, WMCore does not provide run/lumi mask for merge jobs, does it? We just provide the unmerged files to be merged and the framework takes care of it AND creates the FJR, that is then used by the agent for publishing this data and so on. Am I right?
Without a mask the framework would merge all Lumis into the output.

Yes you are right. We don't provide run/lumi mask. @Dr15Jones, how CMSSW gets the run lumi information from unmarked file. (Is the file contains that information?)
The LuminosityBlocks are known internally to the file. There really is no way the framework can misread that information.

Looking at the logs from the jobs, the framework does in deed say the files in question have the same luminosity blocks. The following are excerpts from the merge job cmsRun logs:

--------------------------merge a-------------
04-Nov-2016 11:58:16 CDT  Successfully opened file root://cmsxrootd.hep.wisc.edu//store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/BEA7849C-FDA1-E611-B937-0CC47AD98B90.root
%MSG-w MemoryCheck:  source 04-Nov-2016 11:58:17 CDT PostBeginLumi
MemoryCheck: module source:source VSIZE 550.98 2.00391 RSS 158.156 2.98438
%MSG
Begin processing the 22934th record. Run 1, Event 1225786, LumiSection 7591 at 04-Nov-2016 11:58:17.120 CDT
...
Begin processing the 23256th record. Run 1, Event 1226108, LumiSection 7592 at 04-Nov-2016 11:58:17.302 CDT
04-Nov-2016 11:58:17 CDT  Closed file root://cmsxrootd.hep.wisc.edu//store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/BEA7849C-FDA1-E611-B937-0CC47AD98B90.root
04-Nov-2016 11:58:17 CDT  Initiating request to open file root://cmsxrootd.hep.wisc.edu//store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/F6C514D5-5CA1-E611-857C-003048F5B69A.root
04-Nov-2016 11:58:17 CDT  Successfully opened file root://cmsxrootd.hep.wisc.edu//store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/F6C514D5-5CA1-E611-857C-003048F5B69A.root
%MSG-w MemoryCheck:  source 04-Nov-2016 11:58:17 CDT PostBeginLumi
MemoryCheck: module source:source VSIZE 552.984 2.00391 RSS 159.082 0.464844
%MSG
Begin processing the 23257th record. Run 1, Event 1227078, LumiSection 7599 at 04-Nov-2016 11:58:17.543 CDT
...
----------------------------merge b-----------
03-Nov-2016 22:25:19 EDT  Successfully opened file /mnt/hadoop/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/A685DDD8-5CA1-E611-9020-A0000420FE80.root
Begin processing the 111759th record. Run 1, Event 1225786, LumiSection 7591 at 03-Nov-2016 22:25:19.258 EDT
Begin processing the 112081st record. Run 1, Event 1226108, LumiSection 7592 at 03-Nov-2016 22:25:19.352 EDT
03-Nov-2016 22:25:19 EDT  Closed file /mnt/hadoop/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/A685DDD8-5CA1-E611-9020-A0000420FE80.root
03-Nov-2016 22:25:19 EDT  Initiating request to open file /mnt/hadoop/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/4203A3B8-5CA1-E611-A8FE-24BE05CE4D91.root
03-Nov-2016 22:25:19 EDT  Successfully opened file /mnt/hadoop/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/4203A3B8-5CA1-E611-A8FE-24BE05CE4D91.root
%MSG-w MemoryCheck:  source 03-Nov-2016 22:25:19 EDT PostBeginLumi
MemoryCheck: module source:source VSIZE 547.305 0 RSS 298.508 0.40625
%MSG
Begin processing the 112082nd record. Run 1, Event 1211251, LumiSection 7501 at 03-Nov-2016 22:25:19.431 EDT
Is it possible to get access to the files in question?
How do you know that the files here match the logs for the production?
Chris

Is it possible to get access to the files in question?

I think it is all deleted again. I tried to get it from worker node but wasn't there. And we don't keep the unmerged file some where else. (@amaltaro, Alan, that is correct, right?)

How do you know that the files here match the logs for the production?

We keep track in agent side as well.
To summarize what how the accounting process goes in agent.

Production(Processing as well) job is created with specific job splitting algorithm (mask is passed for each job) - When job is finished, output file info is comes in FWJR (WMAgent FWJR which contains run lumi information for output file from CMSWS FWJR)
We record this in wmbs db.

When merge job is created and submitted the same thing happen.

What we found is basically unmatched record in following file.
/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/A685DDD8-5CA1-E611-9020-A0000420FE80.root
The FWJR for the production job for above output file contains run 1,  lumi 7499, 7500.
But FWJR for the merge job including above file indicates it has run 1, lumi 7501, 7502.
Although it is the same file. I am guessing second information (FWJR from merge job) is incorrect.
But I am not sure how CMSSW gets that information and how the difference could happen.

If you look at the fwjr from  it says lumi section, 7499 and 7500
http://amaltaro.web.cern.ch/amaltaro/forWMCore/Issue_7371/1e19574a-a152-11e6-a6e4-02163e017c3c-749-0-logArchive.tar.gz
<LFN>/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/LHEoutput.root</LFN>
<PFN>LHEoutput.root</PFN>
<Catalog></Catalog>
<ModuleLabel>LHEoutput</ModuleLabel>
<GUID>A685DDD8-5CA1-E611-9020-A0000420FE80</GUID>
<Branches>
  <Branch>LHEEventProduct_externalLHEProducer__SIM.</Branch>
</Branches>
<OutputModuleClass>PoolOutputModule</OutputModuleClass>
<TotalEvents>323</TotalEvents>

<DataType>MC</DataType>

<BranchHash>baad18a51a4cbab54f85db48954b2717</BranchHash>

<Runs>
<Run ID="1">
   <LumiSection ID="7499"/>
   <LumiSection ID="7500"/>
</Run>

but from merge Job fwjr for the same file lumisections are 7591, 7592
http://amaltaro.web.cern.ch/amaltaro/forWMCore/Issue_7371/a7ca0da2-a235-11e6-82a7-02163e017c3c-0-0-logArchive.tar.gz
<InputFile>
<State  Value="closed"/>
<LFN>/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/A685DDD8-5CA1-E611-9020-A0000420FE80.root</LFN>
<PFN>/mnt/hadoop/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/A685DDD8-5CA1-E611-9020-A0000420FE80.root</PFN>
<Catalog></Catalog>
<ModuleLabel>source</ModuleLabel>
<GUID>A685DDD8-5CA1-E611-9020-A0000420FE80</GUID>
<Branches>
  <Branch>LHEEventProduct_externalLHEProducer__SIM.</Branch>
</Branches>
<InputType>primaryFiles</InputType>
<InputSourceClass>PoolSource</InputSourceClass>
<EventsRead>323</EventsRead>
<Runs>
<Run ID="1">
   <LumiSection ID="7591"/>
   <LumiSection ID="7592"/>
</Run>

On Nov 11, 2016, at 9:54 PM, ticoann<EMAIL_ADDRESS>wrote:
The FWJR for the production job for above output file contains run 1, lumi 7499, 7500.
But FWJR for the merge job including above file indicates it has run 1, lumi 7501, 7502.
I believe you mistyped and meant 7591 and 7592 (since that is what is shown in the cmsRun log file).

Although it is the same file.
That is actually an assumption. We do not have any proof it is actually the same file since we can't look at it.

I am guessing second information (FWJR from merge job) is incorrect.
No, that information is correct. The framework reads to LuminosityBlock information from the file. From the actually cmsRun log file output we can see that the framework read the LuminosityBlock as 7591 and 7592 which does not match with what the log file for production said the job which supposedly created that file used as LuminosityBlock numbers.

The most likely scenario to me is either
1) the production log file you pointed me to was not the one that created that file OR
2) the file isn't the file the system thinks it is
Either is pretty scary.
I do not see any sign this is a framework problem as the log files produced by cmsRun match exactly with the FrameworkJobReport for those same logs.
	Chris

No, we don't keep the unmerged files. Once they are merged, there is a cleanup job that comes in and thrown then into the trash bin.

I do not see any sign this is a framework problem as the log files produced by cmsRun match exactly with the FrameworkJobReport for those same logs.

I agree, for the production jobs. However, each of those two merge jobs read these files and in their fjr they report different lumi sections.

I believe you mistyped and meant 7591 and 7592 (since that is what is shown in the cmsRun log file).
Sorry my mistake.  updated. :smile:

The most likely scenario to me is either

the production log file you pointed me to was not the one that created that file OR
the file isn't the file the system thinks it is

I am still confused how this could explain the difference between fwjr from unmarged file and merged file result for exact the same file. (Maybe 2) does.
This is weird since framework reads Lumi information directly from file, both should be the same. (Unless there is uuid collision and same name of the file is create after which is unlikely since we have the file with those lumis).

On Nov 11, 2016, at 10:26 PM, Alan Malta Rodrigues<EMAIL_ADDRESS>wrote:
No, we don't keep the unmerged files. Once they are merged, there is a cleanup job that comes in and thrown then into the trash bin.
I do not see any sign this is a framework problem as the log files produced by cmsRun match exactly with the FrameworkJobReport for those same logs.
I agree, for the production jobs. However, each of those two merge jobs read these files and in their fjr they report different lumi sections.

In all cases the log file output of cmsRun matches the framework job report made by that same cmsRun job. So that backs the claim that the LuminosityBlock recorded in the file being read is what is in the log file.
Therefore the most likely conclusion is the file written by the production job is not actually the same file being read by the merge job.
Do you have the final merge files? In principal we could look at the values for the events in those 'same' LuminosityBlocks and see if they are actually the same events.
Chris

On Nov 11, 2016, at 10:27 PM, ticoann<EMAIL_ADDRESS>wrote:
This is weird since framework reads Lumi information directly from file, both should be the same. (Unless there is uuid collision and same name of the file is create after which is unlikely since we have the file with those lumis).

Looking at the configuration file given to cmsRun, the framework is told to use a very generic name for the output file
/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/LHEoutput.root
something in the workflow management system then renames the file. Therefore we actually have no idea if the file read by the merge job is actually the one produced by the production job.
Chris

The production file is not the same file as used in the merge file. The GUIDs do not match.
From the FJR of the production job we have

/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/GEN-SIM/MCRUN2_71_V1-v1/130000/RAWSIMoutput.root
RAWSIMoutput.root

RAWSIMoutput
006427D9-5CA1-E611-9020-A0000420FE80
but from the FJR of the merge job we have

/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/A685DDD8-5CA1-E611-9020-A0000420FE80.root
/mnt/hadoop/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/A685DDD8-5CA1-E611-9020-A0000420FE80.root

source
A685DDD8-5CA1-E611-9020-A0000420FE80

LHEEventProduct_externalLHEProducer__SIM.

primaryFiles
PoolSource
323

The very first part of the GUIDs do not match
006427D9-5CA1-E611-9020-A0000420FE80
A685DDD8-5CA1-E611-9020-A0000420FE80
Chris

Hi Chris, I think you are looking at the wrong file. GUUI matches with Production job. (A685DDD8-5CA1-E611-9020-A0000420FE80)
http://amaltaro.web.cern.ch/amaltaro/forWMCore/Issue_7371/1e19574a-a152-11e6-a6e4-02163e017c3c-749-0-logArchive.tar.gz
<LFN>/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/LHEoutput.root</LFN>
<PFN>LHEoutput.root</PFN>
<Catalog></Catalog>
<ModuleLabel>LHEoutput</ModuleLabel>
<GUID>A685DDD8-5CA1-E611-9020-A0000420FE80</GUID>
<Branches>
  <Branch>LHEEventProduct_externalLHEProducer__SIM.</Branch>
</Branches>
<OutputModuleClass>PoolOutputModule</OutputModuleClass>
<TotalEvents>323</TotalEvents>

<DataType>MC</DataType>

<BranchHash>baad18a51a4cbab54f85db48954b2717</BranchHash>

<Runs>
<Run ID="1">
   <LumiSection ID="7499"/>
   <LumiSection ID="7500"/>
</Run>

Looking at the configuration file given to cmsRun, the framework is told to use a very generic name for the output file
/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/LHEoutput.root
something in the workflow management system then renames the file. Therefore we actually have no idea if the file read by the merge job is actually the one produced by the production job.

Is that mean actually file is created in generic name and renamed later? Actual file name is created by LHEoutput to GUID in fwjr.
<ModuleLabel>LHEoutput</ModuleLabel>
<GUID>A685DDD8-5CA1-E611-9020-A0000420FE80</GUID>

Isn't the causing the race condition if two jobs putting the files in the same directory? Is that that agent code which rename the file?

On Nov 11, 2016, at 11:02 PM, ticoann<EMAIL_ADDRESS>wrote:
Hi Chris, I think you are looking at the wrong file. GUUI matches with Production job. (A685DDD8-5CA1-E611-9020-A0000420FE80)
http://amaltaro.web.cern.ch/amaltaro/forWMCore/Issue_7371/1e19574a-a152-11e6-a6e4-02163e017c3c-749-0-logArchive.tar.gz

The confusion was there are two files written by the production job, both with very similar GUIDs
/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/LHEoutput.root
LHEoutput.root

LHEoutput
A685DDD8-5CA1-E611-9020-A0000420FE80
and
/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/GEN-SIM/MCRUN2_71_V1-v1/130000/RAWSIMoutput.root
RAWSIMoutput.root

RAWSIMoutput
006427D9-5CA1-E611-9020-A0000420FE80

After talking to Chris, Eric,
I am guessing this is what happens (but I don't have concrete evidence)
For production job - 266543

CMSSW framework copy  the unmerged file with following name (from PSet config which is controlled by PSetTweeks code) - cmssw frame work job report is already created at this point.

/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/LHEoutput.root

WMCore code? renames the output by replacing LHEoutput to GUID from fwjr. - I still couldn't find the code doing this).

In this case there could a race condition there
Other job can writ the the same file  before 2 happens.
/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/LHEoutput.root
Basically first job which wrote to that path is overwritten by second job before it gets renamed.
So first unmerged output file is lost and second unmerged output file is duplicated to different files.
In our case,
/store/mc/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/F08187C8-35A2-E611-ADBF-A0000420FE80.root
and
/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/BEA7849C-FDA1-E611-B937-0CC47AD98B90.root
To fix this, following patch might work.
https://github.com/dmwm/WMCore/commit/84cfa939ee56733e8e6fa2b026ae69889b8b740e
But I can't still find where the step 2 happen which would affect the patch. @amaltaro, do you know where the code resides doing that?

Just to clarify, the framework does not copy the unmerged file. The framework just writes the file in the local directory of the worker node. The copy is done by the workflow system.

The file
/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/LHEoutput.root
never exists. It's a bogus entry in the some intermediate reports that is then supplemented by the stageout code which creates the real file on the mass storage. Are you saying we actually wrote an LHEoutput.root file to mass storage ???

Chris told me that (/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/LHEoutput.root)
is what cmssw framework writes. Not the unmerged file which exists.
(/store/mc/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/F08187C8-35A2-E611-ADBF-A0000420FE80.root)
@Dr15Jones, Chris, did I misunderstood what you told me?

The framework writes `./LHEoutput.root'.

but where? which parameter is it using for current directory?

I'm pretty sure ./LHEoutput.root is written in the pilot scratch area. When it's moved to the storage, it's moved with a UID.

The GUID makes it into the report here
https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/FwkJobReport/FileInfo.py#L72
This is also where we override the report to account for direct stageout to merged.
One comment, this assume that the report already contains
/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/LHEoutput.root
so either @Dr15Jones is wrong about the ./LHEoutput.root or we have yet another piece of wrapper code.

And yes, we write LHEoutput.root into the current working directory, but that doesn't neccessarily mean that the FJR also reports ./LHEoutput.root.

Hmm then, I am not sure how this happens. pilot scratch area is unique to a job, right?
Then where following parameter is used
https://github.com/dmwm/WMCore/blob/master/src/python/PSetTweaks/WMTweak.py#L509

I believe the FJR reports the 'logical' file name as passed to it from the configuration.

Yep, the LFN reported by the FJR has NOTHING to do with where the job puts the file on the local filesystem. It's a LFN that only has meaning for the stageout as a remote directory.

@hufnagel, so PFN below is the current working (job running directory).
/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/LHEoutput.root
LHEoutput.root
and stage out moves the file to directly to below pin, (meaning file name is not renamed after file is copied but before file is copied, correct?)
/mnt/hadoop/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/A685DDD8-5CA1-E611-9020-A0000420FE80.root

No. Current working directory is unknown, no idea what it is. We don't need to know.
/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/
has no meaning whatsoever as far as the worker node file system is concerned. It's a string, nothing more.
That string + GUID IS the stageout location though.
Renamed is the wrong word here. These are all just string manipulations, the first time this corresponds to an actual path and file on a disk somewhere is for the stageout target.

Ie. LFN from the CMSSW FJR does NOT correspond to an actual file somewhere, it's completely bogus. It's just a way for us to forward the UnmergedLFNBase from the workflow configuration to the FJR.

Not sure why this was implemented like this. Before my time. We could also have skipped the LFN completely and in
https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/FwkJobReport/FileInfo.py#L72
constructed the LFN from scratch by doing UnmergedLFNBase+GUID.

Two different pilots never run in the same work space (directory). This is correct statement, right?
So now I don't know how this happened, then. 😞

Correct.

I haven't had time to follow all this, but the LHEOutput.root thing smacks of cmsRun forking off a script to produce the LHE file or something like this. There might actually be code in cmssw that deals with this. IIRC these files are produced in a strange way and @vlimant worked on this.

Ok, so let me summarize:

We have the two processing job logArchive and according to the logs and FJR we wrote out different lumis.

We have the log and FJR of the merge job and according to log and FJR it

a) read in the two files from the jobs in 1) and
b) wrote out different lumis than we had in 1)
Is that correct and all confirmed ?

Hm, so it could be a bug in that code which causes the lumis for one of the jobs in 1) to be reported wrong ? Is that what your are saying ?

The LHEOutput.root file is written by the standard PoolOutputModule. The file format you are thinking about ends with .lhe.

I believe there were two separate merge jobs. Both jobs report having merged the same 2 lumis.

I believe there were two separate merge jobs. Both jobs report having merged the same 2 lumis.

Ok, but one of these merge jobs had an input that was an output of one of the files in 1) and there is a match in file name but a mismatch in lumis ?
If so, sometime between the job in 1) writing it's output and the merge job in 2) the content of the file changed. Which is a bit scary... Apart from the name, does other meta-data match ? Like number of events written by the job in 1) and number of events read by the merge job in 2) ?

Ok, but one of these merge jobs had an input that was an output of one of the files in 1) and there is a match in file name but a mismatch in lumis ?

Yes that is correct. FJR for unmerged file contains lumi [7499, 7500]. but in the merge job FJR for the same file lumi is reported as [7591, 7599].

If so, sometime between the job in 1) writing it's output and the merge job in 2) reading it's input the content of the file changed. Which is a bit scary... Apart from the name, does other meta-data match ? Like number of events written by the job in 1) and number of events read by the merge job in 2) ?

Yes,  event matches. But this is MC (and by definition of splitting algorithm), basically all the jobs has the same events.
The reason I suspect of race condition was the unmerged file from other job which contains [7499, 7500] is located in the same directory and I was confused about cmssw creating the generic name of the file directly to that directory and WMAgent renaming it which doesn't seems to be possible after Dirk pointed out.
Either

something overwrites  the initial unmerged file before merge job ran.
Or FWJ report has wrong information about the input file(unmerged) in the merge job.
I would think it is more plausible 1. case but not sure how could that happen.

@Dr15Jones, Chris told me, that we can actually check what lumis the file contains by examining actual file, right?
Following is the merged file which contains mismatching (between unmerged and merged) lumi case. Can we check whether this actually contains [7591, 7599] or [7499, 7500]?
/store/mc/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/F08187C8-35A2-E611-ADBF-A0000420FE80.root

On Nov 14, 2016, at 10:24 PM, ticoann<EMAIL_ADDRESS>wrote:
Either

something overwrites the initial unmerged file before merge job ran.
Or FWJ report has wrong information about the input file(unmerged) in the merge job.
I would think it is more plausible 1. case but not sure how could that happen.
The log file for the merge job and the FWJ both say the file being read have the unexpected lumi blocks.

@Dr15Jones, Chris told me, that we can actually check what lumis the file contains by examining actual file, right?
Following is the merged file which contains mismatching (between unmerged and merged) lumi case. Can we check whether this actually contains [7591, 7599] or [7499, 7500]?
/store/mc/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/F08187C8-35A2-E611-ADBF-A0000420FE80.root

One can get the list of Run/Lumi/Events in the file by doing
edmFileUtil -e /store/mc/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/F08187C8-35A2-E611-ADBF-A0000420FE80.root
However, my attempt to do so gives this error
Error in TNetXNGFile::Open: [ERROR] Server responded with an error: [3011] Unable to open file /eos/cms/store/mc/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/F08187C8-35A2-E611-ADBF-A0000420FE80.root; No such file or directory
ERR Could not open file root://eoscms.cern.ch//eos/cms/store/mc/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/F08187C8-35A2-E611-ADBF-A0000420FE80.root

Hi everyone,

On Nov 15, 2016, at 10:08 AM, Chris Jones<EMAIL_ADDRESS>wrote:

@Dr15Jones, Chris told me, that we can actually check what lumis the file contains by examining actual file, right?
Following is the merged file which contains mismatching (between unmerged and merged) lumi case. Can we check whether this actually contains [7591, 7599] or [7499, 7500]?
/store/mc/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/F08187C8-35A2-E611-ADBF-A0000420FE80.root

One can get the list of Run/Lumi/Events in the file by doing
edmFileUtil -e /store/mc/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/F08187C8-35A2-E611-ADBF-A0000420FE80.root

edmFileUtil -e root://cmsxrootd-site.fnal.gov//store/mc/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/F08187C8-35A2-E611-ADBF-A0000420FE80.root | grep '(Lumi)' >& lumis.log
I was eventually able to get this to work from CERN. The relevant bit of the output is below:
          1           7496              0            689 (Lumi)
          1           7497              0            690 (Lumi)
          1           7498              0            691 (Lumi)
          1           7591              0            692 (Lumi)
          1           7592              0            693 (Lumi)
          1           7501              0            694 (Lumi)
          1           7502              0            695 (Lumi)

The second column is the LuminosityBlock number and the fourth column is the index in the file to find that lumi. The workflow management system gives the files to merge in what it think is increasing LuminosityBlock number. You can see that '7591' and '7592' are not in the proper order because the workflow system thought the input file being read had luminosity block 7499 and 7500, which it obviously does not.

Ok, this is a bit scary. Question, were both the "fake" lumi 7499/7500 and 7591/7592 files located at the same site ? Ie. is it possible the local storage system gave us the wrong file ? That's about the only thing I can think off. According to all the records we can access from our end, it really looks like we wrote file A from job 1 and then read file A from job B, except it wasn't the same file, even though it had the same name.
Everyone agree with this interpretation ?
If this was indeed at the same site, any chance to ask the site about logs for the two files (creation, access, deletion, general meta-data ?)

Everyone agree with this interpretation ?

Or initial unmerged file is over written after it is created and before merged job runs some how some where. 😄

Everyone agree with this interpretation ?
Or initial unmerged file is over written after it is created and before merged job runs some how some where. 😄

Not a much better option :-).  Either the filesystem gave us the wrong file or someone (not us from what we can tell) overwrote the file with another files content...
Either way, the only way I can see to even have a hope of understanding this is from the mass storage logs (if there are any).

Seems to me like we should find out all we can about the jobs which made the original files

what sites were they produced on?
if the same site, what nodes were they produced on?
-once produced, where were the temporary files moved to?
what site did the merge?
how did the merge job retrieve the files (local file system, xrootd, etc.)?

I have a couple of questions as well.

Can a pilot run multiple jobs when pilot life time is longer then i.e 2 jobs running there?
If 1. is correct statement,
Are jobs running cuncurrently in the pilot?
Are the work space shared between the jobs?
Does workspace is shared?

On Nov 16, 2016, at 09:47, ticoann <notifications<EMAIL_ADDRESS>wrote:
I have a couple of questions as well.

Can a pilot run multiple jobs when pilot life time is longer then i.e 2 jobs running there?

Yes
If 1. is correct statement,
2. Are jobs running cuncurrently in the pilot?
3. Are the work space shared between the jobs?
3. Does workspace is shared?
2 - yes, multiple jobs/pilot
3 and 3 - no, should not be
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/dmwm/WMCore/issues/7371#issuecomment-260980625, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABmErsb4yWbDYLd5FDcysSOq-saleENhks5q-yV4gaJpZM4Kqd7G.

Does workspace is shared?
3 and 3 - no, should not be

I guess then what is the exact mechanism used to separate them?

Workspace is separate for the payloads. Pilot creates different working directories per payload.

But how does the pilot name those directories?

For example /home/glidein_pilot/glide_iHWb9z/execute/dir_25339 (Tier0 job). Last level is payload specific,  there are three of them in this pilot for the three currently running jobs.

And the CMSSW working directory is
/home/glidein_pilot/glide_iHWb9z/execute/dir_25339/job/WMTaskSpace/cmsRun1
every step has it's own working directory.

Assuming after dir_ is the job id, what if the job ID wrapped back around during processing?

It's not the job id, it's some internal pilot counter. GlideInWMS has not idea what is in the payload. And it doesn't wrap around.

Anyways, this is checkable. Job condor logs or maybe even the logArchive will contain the working directory for the job.


host/cli: Use current MariaDB artifact URI when restoring from backup

If this is not done, restores to a newer version of Flynn will fail with:
run-app mariadb error: bootstrap: host error while launching job: "registry: unexpected status 403"'

/cc @josephglanville

LGTM


Convert guides

examples are copied from video.js
guides are copied from video.js and converted to HTML
Readme updated to clarify that no content is edited in this repo

Don't merge yet - need to do some more testing


Tooltip is not closed when exiting enclosing  containing disabled checkbox

Angular 1.4.3, Angular-UI-Bootstrap 0.13.1, Chrome 44.

Put a tooltip on a div
Within that div, put a disabled checkbox:

<input type="checkbox" ng-disabled="true">Label

Run the app.
Hover over the checkbox/label div... tooltip appears.
Exit the div by moving the mouse to the left... tooltip doesn't close.

Note: this is using the standard Bootstrap styling.  You can make the problem less likely to appear by putting a margin (say, 5px) on the left side of the checkbox, but the problem can still occur if you move your mouse fast enough.
Here's a plnkr:  http://plnkr.co/edit/CZpwOw1NXcoI8LQAbnDV

See also related issue #2059.

Forked Plunker to a more useful one here for reference.

This looks like a fundamental issue in jqLite - opened an issue https://github.com/angular/angular.js/issues/12703


2021-11-03 Deposit Together Hindi.md

I have translated the images to Hindi too. I can upload them once you merge.


save data to DB in the new index.php

Hello,
I've read all the previous issues about saving data to sql and json errors, I've also read the numerous harsh replies that were given such as "the script was not intended for that" and so forth;
however,
I simply wanted to save into a DB the name of the uploaded pic, but every bit of PHP code I added to "index.php" (which handles the form) turned into a "SyntaxError: Unexpected token". I tried adding the code in the "handle_file_upload" function, at its end, just before returning the file object.
I would be really grateful if someone helped me, even though the script was intended for "enhancing the client-side file upload".
Thank you a million!

Solved this - I will soon post reply

Hi,
This is how I managed to add uploaded files to a DB, with no "SyntaxError" outputs.
First, add those two functions in the "php/index.php" (newest version of the script) file:
/*start sql insertion here*/
private function query($query) {
	$database = 'database';
	$host = 'localhost';
	$username = 'root';
	$password = '';
	$link = mysql_connect($host,$username,$password);
	if (!$link) {
	   die(mysql_error());
	}
	$db_selected = mysql_select_db($database);
	if (!$db_selected) {
	   die(mysql_error());
	}
	$result = mysql_query($query);
	return $result;
	mysql_close($link);
}

private	function add_img($whichimg) {

// I used this in order to add the pics to a certain user which was previously set with session on login
	$this->user_name = $_SESSION['user_name'];

	$add_to_db = $this->query("INSERT INTO pics (name,tags) VALUES('$whichimg','$this->user_name')") or die(mysql_error());
	return $add_to_db;
}
/*end sql insertion here*/

ALSO, you need to add a line in the "handle_file_upload" function, at the end, BEFORE the last ELSE statement, BEFORE the close-bracket of "if (!$error && $file->name) {" (after "$file->delete_type = 'DELETE';") - the line you have to add is "$file->upload_to_db = $this->add_img($file->name);"
Both variables do not need escaping because it is not an user input - please correct me if I'm wrong.
Hope it helps,
Greets!

Hello,
When I tried the code at the position you mentioned then though I was able to save the file name and other details in database but after saving I get error "SyntaxError: JSON.parse: unexpected character". But if I refresh the page then thumbnails show up i.e. when user starts uploading then it shows this error though images are being saved properly and database insert statement has also executed successfully.. can you please suggest what I am doing wrong?
Thanks

Please upload the code so I can take a look :)

Hello,
Thanks for replying quickly. Please take a look at the index.php file I have mentioned its code below. I am really in need of a help. FYI, I have added my code between the single line comment like "// sunny code starts" and "// sunny code ends"
options = array(
            'script_url' => $this->getFullUrl().'/'.basename(__FILE__),
            'upload_dir' => dirname(__FILE__).'/files/',
            'upload_url' => $this->getFullUrl().'/files/',
            'param_name' => 'files',
            // The php.ini settings upload_max_filesize and post_max_size
            // take precedence over the following max_file_size setting:
            'max_file_size' => null,
            'min_file_size' => 1,
            'accept_file_types' => '/.+$/i',
            'max_number_of_files' => null,
            // Set the following option to false to enable non-multipart uploads:
            'discard_aborted_uploads' => true,
            // Set to true to rotate images based on EXIF meta data, if available:
            'orient_image' => false,
            'image_versions' => array(
                // Uncomment the following version to restrict the size of
                // uploaded images. You can also add additional versions with
                // their own upload directories:
                /*
                'large' => array(
                    'upload_dir' => dirname(__FILE__).'/files/',
                    'upload_url' => dirname($_SERVER['PHP_SELF']).'/files/',
                    'max_width' => 1920,
                    'max_height' => 1200
                ),
                */
                'thumbnail' => array(
                    'upload_dir' => dirname(__FILE__).'/thumbnails/',
                    'upload_url' => $this->getFullUrl().'/thumbnails/',
                    'max_width' => 80,
                    'max_height' => 80
                )
            )
        );
        if ($options) {
            $this->options = array_replace_recursive($this->options, $options);
        }
    }

    function getFullUrl() {
      	return
        		(isset($_SERVER['HTTPS']) ? 'https://' : 'http://').
        		(isset($_SERVER['REMOTE_USER']) ? $_SERVER['REMOTE_USER'].'@' : '').
        		(isset($_SERVER['HTTP_HOST']) ? $_SERVER['HTTP_HOST'] : ($_SERVER['SERVER_NAME'].
        		(isset($_SERVER['HTTPS']) && $_SERVER['SERVER_PORT'] === 443 ||
        		$_SERVER['SERVER_PORT'] === 80 ? '' : ':'.$_SERVER['SERVER_PORT']))).
        		substr($_SERVER['SCRIPT_NAME'],0, strrpos($_SERVER['SCRIPT_NAME'], '/'));
    }

    private function get_file_object($file_name) {
        $file_path = $this->options['upload_dir'].$file_name;
        if (is_file($file_path) && $file_name[0] !== '.') {
            $file = new stdClass();
            $file->name = $file_name;
            $file->size = filesize($file_path);
            $file->url = $this->options['upload_url'].rawurlencode($file->name);
            foreach($this->options['image_versions'] as $version => $options) {
                if (is_file($options['upload_dir'].$file_name)) {
                    $file->{$version.'_url'} = $options['upload_url']
                        .rawurlencode($file->name);
                }
            }
            $file->delete_url = $this->options['script_url']
                .'?file='.rawurlencode($file->name);
            $file->delete_type = 'DELETE';
            return $file;
        }
        return null;
    }

    private function get_file_objects() {
        return array_values(array_filter(array_map(
            array($this, 'get_file_object'),
            scandir($this->options['upload_dir'])
        )));
    }

    private function create_scaled_image($file_name, $options) {
        $file_path = $this->options['upload_dir'].$file_name;
        $new_file_path = $options['upload_dir'].$file_name;
        list($img_width, $img_height) = @getimagesize($file_path);
        if (!$img_width || !$img_height) {
            return false;
        }
        $scale = min(
            $options['max_width'] / $img_width,
            $options['max_height'] / $img_height
        );
        if ($scale > 1) {
            $scale = 1;
        }
        $new_width = $img_width * $scale;
        $new_height = $img_height * $scale;
        $new_img = @imagecreatetruecolor($new_width, $new_height);
        switch (strtolower(substr(strrchr($file_name, '.'), 1))) {
            case 'jpg':
            case 'jpeg':
                $src_img = @imagecreatefromjpeg($file_path);
                $write_image = 'imagejpeg';
                break;
            case 'gif':
                @imagecolortransparent($new_img, @imagecolorallocate($new_img, 0, 0, 0));
                $src_img = @imagecreatefromgif($file_path);
                $write_image = 'imagegif';
                break;
            case 'png':
                @imagecolortransparent($new_img, @imagecolorallocate($new_img, 0, 0, 0));
                @imagealphablending($new_img, false);
                @imagesavealpha($new_img, true);
                $src_img = @imagecreatefrompng($file_path);
                $write_image = 'imagepng';
                break;
            default:
                $src_img = $image_method = null;
        }
        $success = $src_img && @imagecopyresampled(
            $new_img,
            $src_img,
            0, 0, 0, 0,
            $new_width,
            $new_height,
            $img_width,
            $img_height
        ) && $write_image($new_img, $new_file_path);
        // Free up memory (imagedestroy does not delete files):
        @imagedestroy($src_img);
        @imagedestroy($new_img);
        return $success;
    }

    private function has_error($uploaded_file, $file, $error) {
        if ($error) {
            return $error;
        }
        if (!preg_match($this->options['accept_file_types'], $file->name)) {
            return 'acceptFileTypes';
        }
        if ($uploaded_file && is_uploaded_file($uploaded_file)) {
            $file_size = filesize($uploaded_file);
        } else {
            $file_size = $_SERVER['CONTENT_LENGTH'];
        }
        if ($this->options['max_file_size'] && (
                $file_size > $this->options['max_file_size'] ||
                $file->size > $this->options['max_file_size'])
            ) {
            return 'maxFileSize';
        }
        if ($this->options['min_file_size'] &&
            $file_size < $this->options['min_file_size']) {
            return 'minFileSize';
        }
        if (is_int($this->options['max_number_of_files']) && (
                count($this->get_file_objects()) >= $this->options['max_number_of_files'])
            ) {
            return 'maxNumberOfFiles';
        }
        return $error;
    }

    private function trim_file_name($name, $type) {
        // Remove path information and dots around the filename, to prevent uploading
        // into different directories or replacing hidden system files.
        // Also remove control characters and spaces (\x00..\x20) around the filename:
        $file_name = trim(basename(stripslashes($name)), ".\x00..\x20");
        // Add missing file extension for known image types:
        if (strpos($file_name, '.') === false &&
            preg_match('/^image\/(gif|jpe?g|png)/', $type, $matches)) {
            $file_name .= '.'.$matches[1];
        }
        return $file_name;
    }

    private function orient_image($file_path) {
      	$exif = exif_read_data($file_path);
      	$orientation = intval(@$exif['Orientation']);
      	if (!in_array($orientation, array(3, 6, 8))) {
      	    return false;
      	}
      	$image = @imagecreatefromjpeg($file_path);
      	switch ($orientation) {
        	  case 3:
          	    $image = @imagerotate($image, 180, 0);
          	    break;
        	  case 6:
          	    $image = @imagerotate($image, 270, 0);
          	    break;
        	  case 8:
          	    $image = @imagerotate($image, 90, 0);
          	    break;
          	default:
          	    return false;
      	}
      	$success = imagejpeg($image, $file_path);
      	// Free up memory (imagedestroy does not delete files):
      	@imagedestroy($image);
      	return $success;
    }

    private function handle_file_upload($uploaded_file, $name, $size, $type, $error) {
        $file = new stdClass();
        $file->name = $this->trim_file_name($name, $type);
        $file->size = intval($size);
        $file->type = $type;
        $error = $this->has_error($uploaded_file, $file, $error);
        if (!$error && $file->name) {
            $file_path = $this->options['upload_dir'].$file->name;
            $append_file = !$this->options['discard_aborted_uploads'] &&
                is_file($file_path) && $file->size > filesize($file_path);
            clearstatcache();
            if ($uploaded_file && is_uploaded_file($uploaded_file)) {
                // multipart/formdata uploads (POST method uploads)
                if ($append_file) {
                    file_put_contents(
                        $file_path,
                        fopen($uploaded_file, 'r'),
                        FILE_APPEND
                    );
                } else {
                    move_uploaded_file($uploaded_file, $file_path);
                }
            } else {
                // Non-multipart uploads (PUT method support)
                file_put_contents(
                    $file_path,
                    fopen('php://input', 'r'),
                    $append_file ? FILE_APPEND : 0
                );
            }
            $file_size = filesize($file_path);
            if ($file_size === $file->size) {
            		if ($this->options['orient_image']) {
            		    $this->orient_image($file_path);
            		}
                $file->url = $this->options['upload_url'].rawurlencode($file->name);
                foreach($this->options['image_versions'] as $version => $options) {
                    if ($this->create_scaled_image($file->name, $options)) {
                        $file->{$version.'_url'} = $options['upload_url']
                            .rawurlencode($file->name);
                    }
                }
            } else if ($this->options['discard_aborted_uploads']) {
                unlink($file_path);
                $file->error = 'abort';
            }
            $file->size = $file_size;
            $file->delete_url = $this->options['script_url']
                .'?file='.rawurlencode($file->name);
            $file->delete_type = 'DELETE';
			// sunny code starts
			$this->add_img($file->name);
			// sunny code ends
        } else {
            $file->error = $error;
        }
        return $file;
    }

    // sunny code starts
	/*start sql insertion here*/
private function query($query) {
    $database = 'dbname';
    $host = 'localhost';
    $username = 'root';
    $password = '';
    $link = mysql_connect($host,$username,$password);
    if (!$link) {
       die(mysql_error());
    }
    $db_selected = mysql_select_db($database);
    if (!$db_selected) {
       die(mysql_error());
    }
    $result = mysql_query($query);
    return $result;
    mysql_close($link);
}

private function add_img($whichimg) {
    $this->user_name = $_SESSION['user_name']; // I used the following in order to add the pics to a certain user which was previously set with session on login
	$this->user_name = 'hamra';
    $add_to_db = $this->query("INSERT INTO tblphotos (filepath,name) VALUES('".$whichimg."','".$this->user_name."')") or die(mysql_error());
    return $add_to_db;
}
/*end sql insertion here*/

	// sunny code ends
    public function get() {
        $file_name = isset($_REQUEST['file']) ?
            basename(stripslashes($_REQUEST['file'])) : null;
        if ($file_name) {
            $info = $this->get_file_object($file_name);
        } else {
            $info = $this->get_file_objects();
        }
        header('Content-type: application/json');
        echo json_encode($info);
    }

    public function post() {
        if (isset($_REQUEST['_method']) && $_REQUEST['_method'] === 'DELETE') {
            return $this->delete();
        }
        $upload = isset($_FILES[$this->options['param_name']]) ?
            $_FILES[$this->options['param_name']] : null;
        $info = array();
        if ($upload && is_array($upload['tmp_name'])) {
            foreach ($upload['tmp_name'] as $index => $value) {
                $info[] = $this->handle_file_upload(
                    $upload['tmp_name'][$index],
                    isset($_SERVER['HTTP_X_FILE_NAME']) ?
                        $_SERVER['HTTP_X_FILE_NAME'] : $upload['name'][$index],
                    isset($_SERVER['HTTP_X_FILE_SIZE']) ?
                        $_SERVER['HTTP_X_FILE_SIZE'] : $upload['size'][$index],
                    isset($_SERVER['HTTP_X_FILE_TYPE']) ?
                        $_SERVER['HTTP_X_FILE_TYPE'] : $upload['type'][$index],
                    $upload['error'][$index]
                );
            }
        } elseif ($upload || isset($_SERVER['HTTP_X_FILE_NAME'])) {
            $info[] = $this->handle_file_upload(
                isset($upload['tmp_name']) ? $upload['tmp_name'] : null,
                isset($_SERVER['HTTP_X_FILE_NAME']) ?
                    $_SERVER['HTTP_X_FILE_NAME'] : (isset($upload['name']) ?
                        isset($upload['name']) : null),
                isset($_SERVER['HTTP_X_FILE_SIZE']) ?
                    $_SERVER['HTTP_X_FILE_SIZE'] : (isset($upload['size']) ?
                        isset($upload['size']) : null),
                isset($_SERVER['HTTP_X_FILE_TYPE']) ?
                    $_SERVER['HTTP_X_FILE_TYPE'] : (isset($upload['type']) ?
                        isset($upload['type']) : null),
                isset($upload['error']) ? $upload['error'] : null
            );
        }
        header('Vary: Accept');
        $json = json_encode($info);
        $redirect = isset($_REQUEST['redirect']) ?
            stripslashes($_REQUEST['redirect']) : null;
        if ($redirect) {
            header('Location: '.sprintf($redirect, rawurlencode($json)));
            return;
        }
        if (isset($_SERVER['HTTP_ACCEPT']) &&
            (strpos($_SERVER['HTTP_ACCEPT'], 'application/json') !== false)) {
            header('Content-type: application/json');
        } else {
            header('Content-type: text/plain');
        }
        echo $json;
    }

    public function delete() {
        $file_name = isset($_REQUEST['file']) ?
            basename(stripslashes($_REQUEST['file'])) : null;
        $file_path = $this->options['upload_dir'].$file_name;
        $success = is_file($file_path) && $file_name[0] !== '.' && unlink($file_path);
        if ($success) {
            foreach($this->options['image_versions'] as $version => $options) {
                $file = $options['upload_dir'].$file_name;
                if (is_file($file)) {
                    unlink($file);
                }
            }
        }
        header('Content-type: application/json');
        echo json_encode($success);
    }

}

$upload_handler = new UploadHandler();

header('Pragma: no-cache');
header('Cache-Control: private, no-cache');
header('Content-Disposition: inline; filename="files.json"');
header('X-Content-Type-Options: nosniff');
header('Access-Control-Allow-Origin: *');
header('Access-Control-Allow-Methods: OPTIONS, HEAD, GET, POST, PUT, DELETE');
header('Access-Control-Allow-Headers: X-File-Name, X-File-Type, X-File-Size');

switch ($_SERVER['REQUEST_METHOD']) {
    case 'OPTIONS':
        break;
    case 'HEAD':
    case 'GET':
        $upload_handler->get();
        break;
    case 'POST':
        $upload_handler->post();
        break;
    case 'DELETE':
        $upload_handler->delete();
        break;
    default:
        header('HTTP/1.1 405 Method Not Allowed');
}

Hello again,
$this->user_name = $_SESSION['user_name']; // I used the following in order to add the pics to a certain user which was previously set with session on login
$this->user_name = 'hamra';
Do you have a session with the name 'user_name' set? If not, remove that line and just let the next one. If yes (--remove the second line if you want the variable to be equal to whichever the session returns), have you added "session_start();" at the beginning of the index.php file, right after the php open tag?
Tell me whether this solved the issue! Greets! :)

Hello,
Yes, this worked this time. My apologies, I don't know how but I forgot to put session_start on top. But this was unexpected to me, I happened to have seen that PHP does not through any error message or anything in return if the session is not on or if you try to access a variable that does not exists, it just goes empty... but now I feel that PHP was throwing some error message which was being collected as part of Json string.. is that right?
Thanks again :)

Yes :)
Np, I'm glad that it worked :)

Hello Danciu,
You have been very helpful to me in the integration. With your help and a little bit of brainstorming, I am able to save file names in database and submit additional data as well for every image that user uploads. But I am stuck on one thing, I am not sure how to show images after upload from database i.e. after user uploads then at present the system shows images from the folder whereas I want to show user the selected photos by taking there name from database. Can you please guide me?
Thank you very much

Hi again :)
There is a "download" table in the html file in which I removed the Delete button and such, leaving only the file name (you can also leave the thumb), size and an "uploaded" message.
I didn't modify the files in the script furthermore to avoid serious headaches.
So, after the script tables (both upload and download), I created a div which is set to visible with jquery after the upload is complete (...
modify jquery.fileupload.js at the "_onDone" function - line 484 in my script - and add at the end of it your jquery code - mine is:
$("#uploadedpicsh").show();
$("#uploadedpicsh").load('picsmanagerload.php');
...).
In that div you have to call a php script which reads your uploaded pics and everything else you need to do.
Need further help?
Greets!

I am sorry, I could not follow you here. actually I want to use the same interface that jquery has, its good UI and I want to stick to it and by default just show the images by making DB query and showing the records. After this I will customize the delete function too and delete record from DB and then unlink file. For now, I tried to customize the get() function from php/index.php and I did something like this using DB class I have made:-
public function get() {
$file_name = isset($_REQUEST['file']) ?
basename(stripslashes($_REQUEST['file'])) : null;
$sql="select name,filepath from tblphotos ";
$this->query($sql);
$rows=$this->fetchRows();
foreach($rows as $row)
{
if ($row['filepath']) {
$info = $this->get_file_object($row['filepath']);
} else {
$info = $this->get_file_objects();
}
}
header('Content-type: application/json');
echo json_encode($info);
}
but it does not show anything by default. I did extended my DB class to the "UploadHandler"
Please help me.
Thanks

shouldn't you have modified the post function instead? anyway, most of the php in there is json-encoded, do you really prefer getting headaches instead of creating a small php script + jquery function(s) that will work with the upload script?

I can completely understand what you are saying must be correct, but since I am new to this, so asking such silly questions. Okay, so you are saying that I should change the code of post function and the json that it returns is the json code that it displays after file upload.. right? and in this place I can do the select query and fetch records and put them in json format and just return... am I right?
Please help me.

that was just a wild guess, I don't know exactly whether it's that, something else or even something in the JS file - I haven't had the time to go that far, especially that I'm not a JSON expert so I commented some of the index.php file and created my own php that reads the uploaded files and my own jquery script that hides the current pic DIV on delete, refreshes the main div on picture upload etc.
Now that I go through index.php, I see that I commented the following function:
/*private function get_file_objects() {
    return array_values(array_filter(array_map(
        array($this, 'get_file_object'),
        scandir($this->options['upload_dir'])
    )));
}*/

also, the get_file_object function.
Try working around those two first and see what you'll get (imo, a LOT of JSON errors G)

sansphp, I read that you had managed to insert data for each image in mysql.
Could you tell me how you did it, show a piece of code or something?.
I also managed to insert the image name, but when I try to insert a description for each image, all the rows in mysql inserted the data provided for the first image.
thanks

Hi Chaudx, Sure, I will send you detailed reply in about 12hours.
Thanks

thanks, i look forward

Hi Chaudx,
I am sorry for late reply, been really busy these days.
Anyways I have uploaded the whole code folder at http://www.digportfolio.com/multipleuploadamended_uploaded.zip (This link is active for 4days).
To also insert the description with each image I made below mentioned changes:-
a. Change in Javascript in index.php file: The change I made was to add the script to generate textbox corresponding to every image user selects. I did this in script id="template-upload" took a variable with j with value like 'title1', 'title2',.. so on. and with the code line <input name={%=j%} required> I am adding textbox to each image.
b. PHP work in PHP/index.php: In PHP I used session $_SESSION['count'] because while assigning name to textboxes I used the name like title1, title2.. in this case when user uploads lets say 4 pics, since after uploading value of K will not be zero because no refresh is taking place, so I used session similar to variable k in javascript. I am sure there must be another efficient way of doing this as well, but this served my purpose.
c. After saving the next thing was to list the images from database and I managed to do this from php/index.php file by changing a little script in function get_file_objects(), just replace the function scandir($this->options['upload_dir']) with the array of name of images like 'a.jpg', 'b.jpg' i.e. pull names like this from database and rest will work by itself.
I hope this helps. Feel free to let me know if I can help in any other way. I will try to be as quick as I can
Thanks
Sunny

Hi sansphp:
Thanks for the help, great work!
I had something similar, but with common variables. Unfortunately, the problem appeared when you refresh the page.
With your code was the same, but I had not thought of using session variables.
Finally, I managed to do that when you refresh the page $ _SESSION ['count'] back to 0.
Obviously I would not have done it without your help and clear explanation you gave.
thank you very much

Thank you, I am happy it worked for you.

hello you can send me your code? becuase i work so hard so many days to do that but unsuccessful

Hello Vourexx,
Please find below the link to download the setup I already shared before with Chaudx:-
http://www.digportfolio.com/multipleuploadamended_uploaded.zip (Active for coming 3 days)
Let me know if I can help in any other way.
Thanks
Sunny

So now that we know how to save additional details but I am unable to find the way to show those additional details while loading images from database. Showing additional details of the image like Title that user gave to the image, category that user selected, etc. Please help.
Thanks

Hi Sunny,
I am quite new to this JSON stuff, please can you help me with your code cos I want to add description and probably image title to the database image...
You are great... Thanks a billion.
Sabatorio


marginRight of Header Right Button should be 10

react-native 0.40.0, native-base 2.0.4, tested on Android 6
Expected behaviour
The Header>Left is padded left.
But the Header>Right is not padded right.
Steps to reproduce (code snippet or screenshot)
<Header>               <Left/>               <Body><Title>Page 01</Title></Body>               <Right>                 <Button style={{/*marginRight: 10*/}} transparent> <Icon name='ios-menu'/> </Button>                 <Button style={{/*marginRight: 10*/}} transparent> <Icon name='ios-menu'/> </Button>               </Right>             </Header>

sorry. this is my issue. I made some wrong styles.


Add in ReloadDataViewController.

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@c65b9f5). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master      #24   +/-   ##
=========================================
  Coverage          ?   61.82%
=========================================
  Files             ?       28
  Lines             ?      951
  Branches          ?        0
=========================================
  Hits              ?      588
  Misses            ?      363
  Partials          ?        0

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c65b9f5...cbd3692. Read the comment docs.

Just an idea. How about we remove the remove and reset function. and when you tap on a cell it removes the item. and you only need an add button to append a new item into the collection. The item that gets added doesn't have to be a random number. just keep a counter and every time it adds the counter into the collection and increase the counter by one.

This way it also demonstrates the tap handler. which we don't have an example for yet.


Something is funky with 2.2's current worldstrings...

This world is named world, it shows up as WHITE. Whoops.

Fixed a few commits ago.


add flag to disable enhanced address

Pull Request Description
Describe what this PR does / why we need it
Does this pull request fix one issue?

Describe how you did it
Describe how to verify it
Special notes for reviews

Codecov Report

Merging #233 (767c394) into main (189bfe2) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #233   +/-   ##
=======================================
  Coverage   27.41%   27.41%
=======================================
  Files          23       23
  Lines        1674     1674
=======================================
  Hits          459      459
  Misses       1163     1163
  Partials       52       52

Flag
Coverage Δ

unittests
27.41% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 189bfe2...767c394. Read the comment docs.


How to customize an env

Is there any doc or example that I can customize my own env? Hope to get a protocol or an example

I will be updating the README.md with instructions or a example for custom environments over the next few days.

Is there any doc or example that I can customize my own env? Hope to get a protocol or an example

I will be updating README.md with instructions or an example for custom env over the next few days.

By following the guidance provided in README.md, you will be able to create your environment.


add a subcommand 'cargo env' to manage all environment variables

Describe the problem you are trying to solve
To manage all cargo environment variables is inconvenient.
Describe the solution you'd like
To add a new subcommand to manage this.
like:

cargo env, list all maintained and valid variables
cargo env -w envName=envValue, change the value of environment variable

This suggestion comes from my experience in golang, I think it's useful. To look up documentation is inconvenient. Tool is convenient.

That can be built as a third party sub command, if it becomes popular we can look at bringing it in!

If that's true, I'd like to try to do the following:
cargo env --list: Lists all environment variables.
cargo env --envName=envValue: Change the value of envName.
cargo env --del/-d envName: deletes envName;
cargo env --add envName: Add environment variable envName;
cargo env --find envName: search for the environment variable envName;
Anything to add?

@ehuss @alexcrichton Friends, any advice on this?

I don't think there's anything to add over what @Eh2406 already said, this would likely start as an external subcommand and then, if popular, could get integrated into Cargo.

Thanks for the reply. I will try it.


Version 1.4.1 uses MSL 3.2.3

Conflicts resolved manually. Adding PlanarMechanics/Interfaces/PlanarTo3D.mo refused for this version v1.4.1.


removing support metamask in default

Pull Request Test Coverage Report for Build 440

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 79.47%

Totals

Change from base Build 439:
0.0%

Covered Lines:
92

Relevant Lines:
103

💛  - Coveralls