Problem with app markup details

Hi,
I have an issue with checking the details of my app mark details, I have been doing test trade with the account shown below every day, but the last listed transaction is on the 09-05-2019. Is there a delay in this information being parsed, or is this a sign of a problem?
 {
        "app_id": "16261",
        "app_markup": "0.01",
        "app_markup_usd": "0.012991",
        "app_markup_value": "0.01000000000000000000",
        "client_currcode": "GBP",
        "client_loginid": "MX430**",
        "dev_currcode": "GBP",
        "dev_loginid": "MF15***",
        "transaction_id": "435634385",
        "transaction_time": "2019-05-09 13:26:43"
      }
    ]
  },
  "echo_req": {
    "app_id": 16261,
    "app_markup_details": 1,
    "date_from": "2019-04-01 00:00:00",
    "date_to": "2019-05-14 23:59:59",
    "description": 1,
    "limit": 1000,
    "offset": 0,
    "passthrough": {},
    "req_id": 3,
    "sort": "ASC",
    "sort_fields": [
      "app_id",
      "transaction_time"
    ]
  },
  "msg_type": "app_markup_details",
  "passthrough": {},
  "req_id": 3
}

Urggh! Ignore me!! Long day and my daughter is hassling me to play my little ponies... "limit: "1000" was holding back the data.


Error while importing tensorflow from Spyder

I am able to import tensorflow from python terminal outside of Spyder editor but unfortunately, it is showing an error while importing tensorflow package. Error is listed below:

import tensorflow as tf

2022-04-28 21:44:42.471112: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_100.dll
Error in callback <bound method AutoreloadMagics.post_execute_hook of <autoreload.AutoreloadMagics object at 0x000001E4926584C8>> (for post_execute):
Traceback (most recent call last):

  File "C:\Users\suman\miniconda3\envs\TF15\lib\site-packages\IPython\extensions\autoreload.py", line 538, in post_execute_hook
    _, pymtime = self._reloader.filename_and_mtime(sys.modules[modname])

  File "C:\Users\suman\miniconda3\envs\TF15\lib\site-packages\IPython\extensions\autoreload.py", line 184, in filename_and_mtime
    if not hasattr(module, '__file__') or module.__file__ is None:

  File "C:\Users\suman\miniconda3\envs\TF15\lib\site-packages\tensorflow\__init__.py", line 50, in __getattr__
    module = self._load()

  File "C:\Users\suman\miniconda3\envs\TF15\lib\site-packages\tensorflow\__init__.py", line 44, in _load
    module = _importlib.import_module(self.__name__)

  File "C:\Users\suman\miniconda3\envs\TF15\lib\importlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)

  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import

  File "<frozen importlib._bootstrap>", line 983, in _find_and_load

  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked

  File "<frozen importlib._bootstrap_external>", line 728, in exec_module

  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed

  File "C:\Users\suman\miniconda3\envs\TF15\lib\site-packages\tensorflow_core\contrib\__init__.py", line 39, in <module>
    from tensorflow.contrib import compiler

  File "C:\Users\suman\miniconda3\envs\TF15\lib\site-packages\tensorflow_core\contrib\compiler\__init__.py", line 21, in <module>
    from tensorflow.contrib.compiler import jit

  File "C:\Users\suman\miniconda3\envs\TF15\lib\site-packages\tensorflow_core\contrib\compiler\__init__.py", line 22, in <module>
    from tensorflow.contrib.compiler import xla

  File "C:\Users\suman\miniconda3\envs\TF15\lib\site-packages\tensorflow_core\contrib\compiler\xla.py", line 22, in <module>
    from tensorflow.python.estimator import model_fn as model_fn_lib

  File "C:\Users\suman\miniconda3\envs\TF15\lib\site-packages\tensorflow_core\python\estimator\model_fn.py", line 26, in <module>
    from tensorflow_estimator.python.estimator import model_fn

  File "C:\Users\suman\miniconda3\envs\TF15\lib\site-packages\tensorflow_estimator\__init__.py", line 10, in <module>
    from tensorflow_estimator._api.v1 import estimator

  File "C:\Users\suman\miniconda3\envs\TF15\lib\site-packages\tensorflow_estimator\_api\v1\estimator\__init__.py", line 10, in <module>
    from tensorflow_estimator._api.v1.estimator import experimental

  File "C:\Users\suman\miniconda3\envs\TF15\lib\site-packages\tensorflow_estimator\_api\v1\estimator\experimental\__init__.py", line 10, in <module>
    from tensorflow_estimator.python.estimator.canned.dnn import dnn_logit_fn_builder

  File "C:\Users\suman\miniconda3\envs\TF15\lib\site-packages\tensorflow_estimator\python\estimator\canned\dnn.py", line 27, in <module>
    from tensorflow_estimator.python.estimator import estimator

  File "C:\Users\suman\miniconda3\envs\TF15\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 36, in <module>
    from tensorflow.python.profiler import trace

ImportError: cannot import name 'trace' from 'tensorflow.python.profiler' (C:\Users\suman\miniconda3\envs\TF15\lib\site-packages\tensorflow_core\python\profiler\__init__.py)

Warning
It seems you're trying to use a module that doesn't come with our installer. Check this FAQ in our docs to learn how to do this.

Anyone, please help me to fix it.
I am using Windows 11 and the latest Spyder editor.

Hi @SanjuSoni thank you for the feedback. Seems like there is some sort of incompatibilty between the version of tensorflow-estimator and tensorflow-core you are using. How did you install Spyder and tensorflow? Also, you are using Spyder 5.3.0, right? Could you share the conda list of the envs you are using? Let us know

Hii @dalthviz, thanks for pointing out my issue.
The complete information of the spyder editor is

Spyder version: 5.3.0 None
Python version: 3.8.10 64-bit
Qt version: 5.15.2
PyQt5 version: 5.15.6
Operating System: Windows 10

here my OS is windows 11.
==================================
I have created two different environments dedicated to Tensorflow 1.15 and 2.6.0 versions.
The virtual environments are created by conda command and the list of these environments are as follows:
1st environment name ========        TF15             [ tensorflow 1.15 installed here]
2nd environment name========        TF2              [ tensorflow 2.6.0 installed here]
I have installed TensorFlow ver 1.15 in venv TF15 by using the conda command      conda install tensorflow-gpu=1.15
Note: using anaconda kernel, the python interpreter is able to import the TensorFlow properly in both environment. I am using Spyder editor from last two years and it acts like a universal editor for me in case of python scripts.
But, unfortunately, after reinstallation of my OS and spyder version 5.3.0, I am facing such issue.
Kindly provide some alternate/solution for it.
Thanks!!!

@SanjuSoni you will need to manually downgrade tensorflow-estimator to 1.15 (not sure why but conda install tensorflow-gpu=1.15 installs tensorflow-estimator 2.6.0 which I would say is incompatible with tensorflow 1.15.0). For that, run from an Anaconda prompt with your env activated conda install tensorflow-estimator=1.15
This is issues is not directly related with Spyder but let us know if the above helps

You are Genius @dalthviz SIr.
It works for both environments TF15 and Tf2 after degrading the tensorflow-estimator.
In TF15 env
import tensorflow as tf
2022-04-29 11:48:54.814574: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_100.dll
tf.version
Out[2]: '1.15.0'
=========================
In TF2 env
import tensorflow as tf
tf.version
Out[2]: '2.6.0'
===========================
Thanks for your valuable support!!!!!!!!

Using the command conda install tensorflow-estimator=1.15 crashed python on my system.
Package details are as follows:
cudatoolkit     10.0.130
cudnn    7.6.5
cython  0.29.32
h5py    2.10.0
hdf5  1.10.4
keras    2.1.3
keras-applications 1.0.8
keras-preprocessing   1.1.2
numpy  1.18.1
numpy-base  1.21.5
opencv-contrib-python   <IP_ADDRESS>
opencv-python   <IP_ADDRESS>
opencv-python-headless   <IP_ADDRESS>
pillow    9.2.0
python        3.7.13
scikit-image    0.15.0
scipy   1.7.3
shapely 1.7.0
tensorboard   2.8.0
tensorflow   1.15.0
tensorflow-estimator 1.15.1
tensorflow-gpu    1.15.0

@parthkvv seems like you have tensorboard 2.8.0 which is not following the rest of your related tensorflow packages (which have version 1.15.x)


WHERE IS DOWNLOAD LINK??????????

download link cannot be found, quick link has 404'd

Press the green CODE button the download zip.


  They are always underfoot and a nuisance, like flying insects.  I don't know what they're saying and I don't care, I've got work to do.  I've turned off litterally EVERYTHING in suggestions, and everything else that seemed to have something to do with them.  Somewhere there was a setting for how long to wait before putting up a tooltip; I set it to an astronomically high number.  But still they keep on popping up.  I've been programming in JavaScript for 15 years, I don't need a 14 square inch box explaining console.log(), obstructing where I want to click.  Is there some setting I can use to just turn them all off?  Is there some setting that must still be on that I've missed?

VS Code version: Code 1.76.2 (ee2b180d582a7f601fa6ecfdad8d9fd269ab1884, 2023-03-14T17:55:54.936Z)
OS version: Windows_NT x64 10.0.22621
Modes:
Sandboxed: No

System Info

Item
Value

CPUs
AMD Ryzen 7 PRO 5850U with Radeon Graphics      (16 x 1896)

GPU Status
2d_canvas: enabledcanvas_oop_rasterization: disabled_offdirect_rendering_display_compositor: disabled_off_okgpu_compositing: enabledmultiple_raster_threads: enabled_onopengl: enabled_onrasterization: enabledraw_draw: disabled_off_okskia_renderer: enabled_onvideo_decode: enabledvideo_encode: enabledvulkan: disabled_offwebgl: enabledwebgl2: enabledwebgpu: disabled_off

Load (avg)
undefined

Memory (System)
27.83GB (14.22GB free)

Process Argv
/home/v-abonadio/.profile /home/v-abonadio/.profile_common /home/v-abonadio/.profile_common~ /home/v-abonadio/.profile_cygwin /home/v-abonadio/.profile_cygwin~ /home/v-abonadio/.profile_linux /home/v-abonadio/.profile_linux.ojibwe /home/v-abonadio/.profile_linux.owikeno /home/v-abonadio/.profile_mac /home/v-abonadio/.bashrc /home/v-abonadio/.bashrc_common /home/v-abonadio/.bashrc_common~ /home/v-abonadio/.bashrc_cygwin /home/v-abonadio/.bashrc_linux /home/v-abonadio/.bashrc_linux.ojibwe /home/v-abonadio/.bashrc_mac --crash-reporter-id b1a29340-5506-4d99-90c3-927dd01918a0

Screen Reader
yes

VM
0%

Extensions: none
A/B Experiments
vsliv368:30146709
vsreu685:30147344
python383:30185418
vspor879:30202332
vspor708:30202333
vspor363:30204092
vslsvsres303:30308271
pythonvspyl392:30443607
vserr242cf:30382550
pythontb:30283811
vsjup518:30340749
pythonptprofiler:30281270
vshan820:30294714
vstes263cf:30335440
vscorecescf:30445987
pythondataviewer:30285071
vscod805:30301674
binariesv615:30325510
bridge0708:30335490
bridge0723:30353136
cmake_vspar411:30581797
vsaa593cf:30376535
pythonvs932:30410667
cppdebug:30492333
vsclangdf:30486550
c4g48928:30535728
dsvsc012cf:30540253
pynewext54:30695312
azure-dev_surveyone:30548225
pyindex848:30662994
nodejswelcome1:30587005
282f8724:30602487
pyind779:30671433
89544117:30613380
vscrp:30673768
pythonsymbol12:30671437
a9j8j154:30646983
vsccsb:30677849
vscodedisable:30660115
defaultazurewalk:30687958
pythonms35cf:30686773

You can turn off the editor hover by setting "editor.hover.enabled": false, parameter hints that call out what arguments do can be turned off with "editor.parameterHints.enabled": false

wow, thanks for the personal response!
My settings.json has no editor.hover anything, the only hover is css.hover.documentation,which is false.
Also no parameterHints, only editor.suggest.showTypeParameters, which is set to false.
I'm using vs code "1.77.3 (user setup)".  I'd copy & paste the whole version block but it's on a different machine and I don't know how to copy/paste between mac & windows 🙄
and this seems to be AppData/Roaming/Code/User/settings.json
Allan Bonadio j
@.***

On May 2, 2023, at 9:29 AM, Daniel Imms @.***> wrote:
You can turn off the editor hover by setting "editor.hover.enabled": false, parameter hints that call out what arguments do can be turned off with "editor.parameterHints.enabled": false
—
Reply to this email directly, view it on GitHub https://github.com/microsoft/vscode/issues/178455#issuecomment-1531771246, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHXLIETARQLBXGWIPEAJKDXEEY6TANCNFSM6AAAAAAWJXESSY.
You are receiving this because you authored the thread.

You need to add the setting if you want to disable them, they're definitely there as part of the core editor:

Or via the settings editor:


add TLFIdentifyBehavior_PAGES

If it isn't fixing anything, then I'd prefer not to add yet another identify behavior.
The chat GUI has very specific handling of identify/track failures, hence its special case here.  But I would be wary to let others start to skip failures.
I don't really know what identify/track has to do with keybase pages, so if there's something I'm missing, please let me know.

@patrickxb OK thanks for the input! I'll close this and the KBFS one for now.

I don't really know what identify/track has to do with keybase pages, so if there's something I'm missing, please let me know.

I think if the Pages bot doesn't follow any user, then if their proofs break it won't stop kbpagesd from reading from KBFS. So the user would still be able to fix their Pages hosted proof.


Fixing a problem in helm configmap

If / else chain was broken because of a {{end}} in the way.
helm lint was stating that with the following error:
[ERROR] templates/: parse error in "trickster/templates/configmap.yaml": template: trickster/templates/configmap.yaml:53: unexpected {{else}}

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Do not expose this tool on any networks

Describe the bug
I own/operate dedicated infrastructure, I had a virtualizor VM running Ubuntu 22.04 Jammy, I did have this application exposed to the internet. However, I received a notification from the datacenter on an attack coming from that VM. I instantly took a peak and saw the VM used almost 1TB of bandwidth in the last day.
 If someone wants to investigate I can provide the VM creds.
The only two things on the VM was Ubuntu 22.04 Jammy + Bolt, nothing else, it was a fresh VM specifically for bolt.
Link to the Bolt URL that caused the error
I don't have one
Steps to reproduce

Install Ubuntu 22.04 Jammy and Bolt and expose with --host

Expected behavior
Not to go rogue
Screen Recording / Screenshot
No response
Platform
Ubuntu 22.04 Jammy
Additional context
No response

Your going to need a lot more logs, and screenshots also just stating you have a VM in a datacenter = you can get hacked in 5 minutes.  It's got nothing to do with Bolt or Ubuntu - your in a public datacenter.

Your going to need a lot more logs, and screenshots also just stating you have a VM in a datacenter = you can get hacked in 5 minutes. It's got nothing to do with Bolt or Ubuntu - your in a public datacenter.

Again, I am including someone who works on Bolt access to the VM/image and any logs they need. I am not going to publicly broadcast this stuff here, that would be dumb.
However, its extremely high chance that Bolt has something to do with it given there is plenty of VMs on 22.04 running on same build.

Depends on what API calls your making also and how many, and if running ollama what your downloading

I see nothing in bolt code that can do anything you describe:

I'm thinking of closing this issue because I don't think this is related to the project really as @syndicate604 is saying. But please provide more details if you have that @Shybella


[Improve] [Flink-Kubernetes-V2] Change doTriggerSavepoint log

What changes were proposed in this pull request
Change doTriggerSavepoint log

Brief change log

Verifying this change

This change is a trivial rework / code cleanup without any test coverage.
(or)
This change is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:

Does this pull request potentially affect one of the following parts

Dependencies (does it add or upgrade a dependency): (yes / no)
no

cc @wolfboys @Al-assad


Remove unused templates

First iteration on Andrew's list of unused templates. Removed the obviously unused templates.
This was first created for the issue #485
Are there any concerns to addressed further before or after merging this PR?
The full list of the change in this issue #513
I thought we could merge this, remove this branch, then complete work in a branch for #513
Mentions?
@andrewtpham @ninetteadhikari
Issue(s) affected?
#485

@ninetteadhikari
@Kimaiyo077 is taking care of resolving the conflicts. We quickly went over it.


Remove backgrounds of lanes + version update(s)

Not sure about do we need this or not :D
Also changed version to 1.91 to allot Tamper/Grease to update.
Furthermore I have tried to remvoe the moving clouds from the main screen, but I am not enough for that :P
Can you look at it?

I don't think this is needed. Also I'm not even sure what you're trying to accomplish?


can this library use in X86 software

can you kindly give some examples in X86

Yes, it works on x86 (as well as other architecture, e.g. ARM, ARM64, WebAssembly). Here's a trivial example:
#include <fp16.h>

float round_to_fp16_and_back(float x) {
  // Convert x to IEEE FP16
  uint16_t x_as_fp16 = fp16_ieee_from_fp32_value(x);
  // Convert back to float (IEEE FP32)
  return fp16_ieee_to_fp32_value(x_as_fp16);
}


[bug] Double-quotes on CssJs.unsafe for font-family that include quotes?

Hi David!
Our team has been using quotes for font family strings that have whitespace:
  font-family: "Inter Semi Bold";

This seems like it should be valid, maybe single quotes?, but produces:
      CssJs.unsafe("fontFamily", ""Inter Semi Bold""),

I have removed the quotes for now but maybe something to be handled here?
Thanks
Alex
[edit: this on 0.36.2]

Hey @AlexMoutonNoble, that's a bug for sure. It should generate CssJs.unsafe("fontFamily", {js|"Inter Semi Bold"|js}).

Hey @AlexMoutonNoble, what's the annotation you use for font family in this case? (cx, styled, global? and probably you are using ReScript, right?

Fixed in 7f0594d2b8dcc2c86753fa699bd32a652bcdfe22


Reasoning behind field-boolean-null

I am wondering why do you recommend BooleanField instead of NullBooleanField?
I was always recommending NullBooleanField 🙂

It was

likely to be deprecated in a future version of Django

since version 2.1 https://docs.djangoproject.com/en/2.1/ref/models/fields/#nullbooleanfield
and in version 3.1 finally became deprecated https://docs.djangoproject.com/en/3.1/ref/models/fields/#nullbooleanfield.
I should document that now the check make sense only for django<3.1.

Oh, thanks! I guess that it will be helpful for new users to have 'reasoning' section for all checks.
It could possibly teach a lot of people something good! 👍

Oh, thanks! I guess that it will be helpful for new users to have 'reasoning' section for all checks.
It could possibly teach a lot of people something good!

If only I had time :stuck_out_tongue_winking_eye:


Export not working

I requested the export of data from a municipality in pbf format and after eight hours the file was not generated.
Is this common or is it a bug?
url: https://export.hotosm.org/en/v3/exports/5faf3f08-901e-446a-aace-6302865cf598

Hi ! Thank you for the report , Problem has been acknowledged and developer is working on fix !

Fixed with the PR !


Major speedup of floating-point handling in JSON/Text

Previously, Float and Double handling for JSON/Text were handled with
the Double(String), Float(String), String(Float), and
String(Double) initializers.  This was very slow, with a lot
of time spent in constructing String temporaries and translating those
temporaries to and from UTF8.
To decode a float, for instance, we would convert the UTF8 into
a String, then the Swift Standard Library would convert the string
back into UTF8 in order to use the C library strtof function
internally.
This code cuts out the middleman: We pass the UTF8 bytes directly to
the C library functions strtod and vsnprintf ourselves, avoiding
the intermediate String objects entirely.
For decode, this is a 10x improvement, putting float/double
parsing much more nearly on par with integer handling.
For encode, this is a big (4x) improvement, but we still have
a long way to go.  Invoking vsnprintf from Swift requires using
Swift's expensive CVarArg facility; profiling shows about 50% of
our total time being spent just to marshal varargs.
Actual results: After modifying the perf_runner test to write out
numbers with fractional parts, I now see the following results for
encoding/decoding 100 float fields to/from Text format (JSON results
are similar):

Before: 600 µs Text format encoding, 260 µs decoding
After: 140 µs Text format encoding, 22 µs decoding
C++: 45 µs Text format encoding, 36 µs decoding

For comparison, encoding 100 int64 fields using the SwiftProtobuf text
encoder takes about 12µs on the same hardware.

Hoping this doesn't sound too critical—
Making things faster is a good goal, but it feels like we're going even further down the path of "assembly language with Swift syntax" that I mentioned before with these changes. Are we sacrificing readability and maintainability of the runtime library by doing this? Are we going to be able to look at this code in 6 months and understand it easily?
When I made the original JSON performance boost, it was still very much idiomatic Swift—it used the high-level string view APIs, indexes, and so forth—but as time goes on, more and more of the runtime library looks less and less like Swift. The readability and design of the code has taken a distant back seat to squeezing out extra microseconds, and I wonder if we need to rebalance these priorities.
For binary encoding, I'm willing to concede that dropping into unsafe-land to squeeze out performance gains is probably something we need to do, since the whole point of the encoding is to be small and fast. But for text and JSON, if we've reached the point where we're mostly passing around raw UTF-8 pointers and calling into C functions, what's the point of writing it in Swift at all? We could get even better performance by implementing the whole thing in C and wrapping it in a Swift interface. (Obviously I don't want to do this.)
Another fear I have is that we're setting the efficiency bar impossibly high, and when the time comes that we want to revisit parts of the runtime design and improve them, we'll hesitate because they might result in a slight runtime increase in the raw numbers, even if those numbers would still be "great" by an objective standard.
We're already matching or beating C++ in text/JSON performance metrics, so is it worth going farther beyond that when we end up paying for it in code readability?

Hoping this doesn't sound too critical...

Although motivated by performance, this particular PR is also a step towards developing UTF-8-oriented support code that I think will make the JSON and Text logic easier to understand.  Specifically, I've created a DoubleFormatter that hides details of converting between UTF-8 and Float/Double.  Isolating those details made it feasible to explore using C library functions that happen to be much better suited to UTF-8-oriented operations.  My description above explains this in holistic terms:  How the compiled form now works after everything gets optimized down.  But I think the source is actually clearer than before.
Swift's standard String support is heavily biased towards UTF-16 and has a lot of compromises in its UTF-8 handling right now.  JSON and Text formatting and parsing are inherently UTF-8 based, and it's not yet entirely clear to me how best to express UTF-8 operations in Swift.  Today, we're using a mix of techniques: JSON/Text encoding relies heavily on Array<UInt8>, while decoding is using simple pointers a lot.  Two easy improvements I hope to work on soon:  migrating a lot of the simple pointers to buffer pointers, and developing tailored types to wrap key concepts (NameMap.Name and DoubleFormatter are two steps in that direction).  I expect that future Swift evolution will provide even better tools.

We could get even better performance by implementing the whole thing in C ... (Obviously I don't want to do this.)

Nor do I.  But I'm not averse to selectively using small bits of C where C APIs happen to be a better fit for the task at hand.  Parsing floats and doubles from UTF-8 is one of those cases; strtod is almost exactly the right API for that.  Formatting floats and doubles into UTF-8 may be another, though vsnprintf is not the most elegant API around, and the CVarArgs tax is higher than I expected.  At least the current PR manages to isolate that operation so we can continue looking for better ways to express it.

Another fear I have is that we're setting the efficiency bar impossibly high...

My first focus for Swift Protobuf was correctness.  As I learned what that involved, I wrote a lot of ugly code filled with special cases that eventually got refactored into a clearer form that better expressed the real problems at hand.
Now that we have a rich test suite to help protect correctness, I've shifted my attention to performance.  As before, eliminating unnecessary overhead has introduced a few awkward workarounds, but as I better understand what work is really needed (and conversely, what work can be avoided), I plan to continue looking for clearer ways to express the code.  And yes, I do expect some of that refactoring to yield ground on performance.
As long as we do continue to look for ways to improve the clarity of the code, I'm confident we'll find them.  And now that we've managed to hide the decoding and encoding internals within the library, I think we'll have the luxury post-1.0 of continuing to refactor our internal logic in fairly dramatic ways without impacting our clients.

Side comment -
One thing I've always sorta liked in the C++ impl is the CodedInputStream/CodeOutputStream models.  It allows the library to wrap something that writes to buffers, file descriptors, etc. meaning they can read/write directly with files, networking, etc. and not just in memory.
In our quest for performance, we've done all decoding/encoding around having the full payload in memory.  Which while ok for clients, might hurt things from a server usage pov.  I've got some ideas about how to speed out binary serialization, but they would never work in a streaming model, so I'm hesitant to actually put effort in to them.
Anyway, my fear is are we optimizing to the point that if we ever need to support streaming for server usage, we'll have to duplicate lots of code and/or accept what appears to be large performance hits because things were pushed too specific initially.
I haven't looked in detail, but the C++ has a few methods with NoVirtual in the names and have very carefully crafted things so buffer only stuff can still be really fast, but still support the reading/writing external things.  There might be similar things we can do to split out specializations for in memory vs. more generic.

migrating a lot of the simple pointers to buffer pointers

Coincidentally, I have a branch sitting around that does exactly that for the JSON decoder. Unfortunately a couple PRs that have gone in since I started, along with this one, will make merge conflicts numerous 😕
By moving to buffer pointers, my hope is that we can actually rewrite the decoder to be expressed in terms of Sequence instead. (We'd have to eliminate the backtracking that happens as we tokenize, but we're just talking basic DFAs and would end up improving performance slightly anyway.) Since buffer pointers provide that conformance we'd get that for free, but then it would also give us the ability to decode streaming data sources that can also be expressed that way, as @thomasvl mentions.

Swift's standard String support is heavily biased towards UTF-16 and has a lot of compromises in its UTF-8 handling right now. JSON and Text formatting and parsing are inherently UTF-8 based, and it's not yet entirely clear to me how best to express UTF-8 operations in Swift.

I'm not sure it's any more biased toward or against UTF-16 than UTF-8; the _StringCore type used internally is written to efficiently handle either UTF-8 or UTF-16 encoded text and—at least if we start from an empty string and start building an encoding result up—if we never append anything that isn't ASCII, it will never shift to the UTF-16 representation.
As for decoding, that does get trickier because it depends where the string comes from. If it's a pure Swift string that contains text-format or JSON, then there's a chance that it's already in the UTF-8-only core representation, but if it's wrapping a Foundation NSString that came from, say, a file API, it's much harder to reason about its representation.
Part of the problem is that the UTF-8 and UTF-16 views on String aren't mutable, so there's no convenient way to build up a String by appending UTF-8 code units. If anything, I would say that String is biased toward Unicode scalar representation, because that's the only view on String that implements the full set of mutating collection operations.
All that being said, let's go ahead and move forward with this. But let's also keep an eye on opportunities like what @thomasvl mentioned about streaming possibilities; the binary coding could definitely benefit from being able to support that model for encoding and decoding, so figuring out the right way to express it generally in terms of another protocol without assuming the whole thing is in memory and using pointers throughout would be very beneficial. For JSON and text it would also be nice, but I think it's not as critical there.


Microbit radio

Hi! This code is correct? https://lancaster-university.github.io/microbit-docs/ubit/radioevent/#ubitradioevent
I cant use the radio with this.
I am interesting to manage radio without ubit.init(), any idea?
Thanks!

Yes, that is also implemented in CODAL, have you tried it already? Are there any issues when it's running?

Hi! if you copy and paste the code of the page, you dont receive messages. I dont know if the problem is for me, because i use CODAL and maybe this code is for DAL.


travis should run tests in all environment combinations

Travis should run the tests in all the environment combinations. Plus it'd be nice to have a way to make sure tox.ini and .travis.yml are in sync.

Worst case, it could run a pared-down tox config that would take care of different pip versions, while Travis itself took care of Python versions. That's essentially what http://jsatt.com/blog/using-tox-with-travis-ci-to-test-django-apps/ does.

This should solve all our problems: https://github.com/msabramo/python-panci. We don't even necessarily have to use it; following the pattern of its multi-python-version example travis configs might be enough.


dart2js and analyzer disagree on valid mixins

The program below, when analyzed with the analyzer gives an error only the second mixin application, whereas with dart2js there is an error on both.  This is without supermixins turned on.
They both look equivalent to me so I would expect that both should be an error, but maybe there is a subtle distinction with mixin application classes?
cc @bwilkerson @lrhn @munificent @vsmenon
class Base { }

class Mixin2 {}
class Mixin3 {}

class Mixin1 = Object with Mixin2, Mixin3;
// Error on dart2js, not on analyzer
class A extends Object with Mixin1 {}

class Mixin4 extends Object with Mixin2, Mixin3 {}
// Error on both
class C extends Object with Mixin4 {}

void main () {
  new C();
  new A();
}

Both should be errors. Without the supermixins flag, you are not allowed to derive a mixin from a class with a superclass other than Object.
The superclass of Mixin1 is Object with Mixin2. The superclass of Mixin4 is Object with Mixin2, Mixin3. Neither has Object as superclass.

Thanks.  Your answer is illuminating - it explains why both the analyzer and the dart vm (with supermixins enabled) think that, in the code below, mix3 is the only concrete method in A and C.
Can you elaborate on how this falls out of the spec language?  I did give the spec a reasonably close reading and wasn't able to figure out if the behavior on the example below was correct or not.
class Base { }

class Mixin2 { void mix2() {print("mix2");} }
class Mixin3 {void mix3() {print("mix3");}}

class Mixin1 = Object with Mixin2, Mixin3;
// No error
class A extends Object with Mixin1 {}

class Mixin4 extends Object with Mixin2, Mixin3 {}
// Error
class C extends Object with Mixin4 {}

void main () {
  try {
    new A()..mix2();
  } catch(e) {
    print(e);
  }
  try {
    new A()..mix3();
  } catch(e) {
    print(e);
  }
  try {
    new C()..mix2();
  } catch(e) {
    print(e);
  }
  try {
    new C()..mix3();
  } catch(e) {
    print(e);
  }
}

There is a mix3 in A and there shouldn't be a mix3 in C. The mixin derived from C is empty (it's the declarations of the {} body).
Dart allows you to create a class in two ways:
class C1 extends S { members }
class C2 = S with M;

Both of these creates a new class by extending the superclass S with a mixin containing instance method declarations. The former has a literal mixin (the { members } part of the declaration, which can also contain static members on top of the mixin members) and the latter has a named (derived) mixin.
(I'll just ignore interfaces here).
You derive a mixin from a class (which is a superclass with a mixin on top) by ignoring its superclass and just taking the declarations of the mixin.
When we write:
class A extends B with C, D { methods }

it's a composite mixin application that can be separated into individual mixin applications as:
class $BC = B with C;
class $BCD = $BC with D;
class A extends $BCD { members }

So, for the example above:
class Mixin1 = Object with Mixin2, Mixin3;

declares a class Mixin1 with superclass Object with Mixin2 and mixin Mixin3. When you derive a mixin from that, you get the same mixin as the one derived from Mixin3 (which contains mix3).
class Mixin4 extends Object with Mixin2, Mixin3 {}

declares a class Mixin4 with superclass Object with Mixin2, Mixin3 and mixin {}. When you derive a mixin from Mixin4, you get {}.
Because of that, only new A()..mix3(); should succeed, the rest are calling methods that are not declared by A or C.
This is all in the "Mixins" section of the spec. It actually starts out fairly clear and conceptual, before it gets into details about mixin application.

This is all in the "Mixins" section of the spec. It actually starts out fairly clear and conceptual, before it gets into details about mixin application.

I beg to differ.  :{
Thanks for the walk through, I think that gave me the key to how to interpret the spec.  The essential bit is that this:
The effect of a class definition of the form class C = M; or the form class
C < T1, . . . , Tn > = M; in library L is to introduce the name C into the scope
of L, bound to the class (10) defined by the mixin application M

is a back reference to
A mixin application of the form S with M; defines a class C with superclass
S.
A mixin application of the form S with M1, . . . , Mk; defines a class C whose
superclass is the application of the mixin composition (12.2) Mk−1 ∗ . . . ∗ M1 to
S.
In both cases above, C declares the same instance members as M (respectively,
Mk).

the key point being that the members of M (respectively, Mk) are the members of the defined class, and hence are present in the implicitly defined mixin.

The former has a literal mixin (the { members } part of the declaration, which can also contain static members on top of the mixin members) and the latter has a named (derived) mixin.

This seems like a useful way of thinking about it, but it's definitely not present in the spec.
Overall, I feel like the language in this section could be improved...


configuration: filter unroutable addresses on Android by default

Description: After experimenting with it in Lyft apps we saw a positive impact of enabling filtering. It's enabled to all Lyft users who have EM enabled.
Risk Level: Low, experimented with.
Testing: N/A
Docs Changes:  Done
Release Notes: Done

@alyssawilk Updating this so that it matches what we use internally at Lyft :)


Switch to use str instead of number for colab_url

Fix nightly failure:
File "tensorflow/tools/ci_build/update_version.py", line 253, in colab_url
version_string = "%d.%d.%d" % (version.major, version.minor, version.patch)
TypeError: %d format: a number is required, not str

I believe @case540 fixed this internally as well.


[WIP] Fix wallet connection issues on mobile with appkit

Changes
This adds some rudimentary support for Appkit (replacing the existing logic of rainbowkit). I modified the code so that a click on the 'Confirm' button will trigger a request for signing. The input amounts don't matter and it will always be the same transaction to sign. This is done to facilitate speedy testing of the signing functionalities.
Before merging the changes, we would need to adjust the design of the connect button etc. and remove the temporary logic for testing.
Closes #258.
Findings
I did some tests and found that using the latest dependencies of wagmi/viem done in #268 in combination with Appkit works quite well. To elaborate, I found that the 'No matching key. History' error did not occur with Appkit ever. Neither for connecting the wallet nor for disconnecting nor when signing a message. Tested with and without an app refresh in between.
It's still not perfect though. I noticed that signing requests don't always trigger the respective app. For example, when I connect to the prototype with WalletConnect and Zerion on my phone, the connection to the account works well. Zerion opens, let's me choose my account, etc. However, when a transaction signature is requested, Zerion never opens automatically. It just seems as if nothing happens to the user. When I open Zerion after the signing is requested (and it seems like nothing has happened) I can see the signing request in Zerion though.
For Metamask it's similar but a little different. The wallet connection with Metamask works well bug when it comes to signing a transaction, sometimes I see an intent where I can open Metamask and sometimes I don't. When I don't see the intent to open the app, again, I can manually open Metamask and will see the request for signing.
I found a related bug ticket https://github.com/reown-com/appkit/issues/1504 and it seems like this is not only an AppKit problem but also something related to wagmi.
As a side note, I also found another bug ticket https://github.com/reown-com/appkit/issues/2076 mentioning that automatic redirect back to the calling application is not possible on iOS 17. This worked on my Android phone when using Metamask.
Outlook
All in all, I think this is an improvement over the connection with rainbowkit. I tested the current version on polygon-prototype-staging again (which has rainbowkit + the latest dependencies of wagmi/viem) and I still encountered the 'No matching key. History' and other similar errors. This didn't happen to me with Appkit.
Thus, I would suggest we try to deal with the shortcomings on Appkit, possibly showing a notification to the user when a signing is requested that if they don't see any action appearing on screen, they should check their connected wallet application manually. It's not a great UX but at least it works technically.

Tested this:
Setup:
Motorola edge 30 ultra. Chrome + Metamask
First attempt:
I connected a wallet and it consistently worked, meaning click on confirm prompt me to move to Metamask and sign.
Forcing a failure:
Nevertheless, I then disconnected from my wallet on Vortex, connected again by creating a new account on Metamask.
After this, clicking on confirm was consistently "failing", yet in Metamask I could see the request.
After a few attempts at changing back again to my first wallet (which was failing many times but finally succeeded), I changed back and forth from my first address and my second. Seems to be that always with my second address, the prompt to Metamask would not show while with the first it would.

Seems to be that always with my second address, the prompt to Metamask would not show while with the first it would.

That's an interesting observation 🤔 Could still be that it's a coincidence. As for me, I always tested with the same account and sometimes the prompt worked, sometimes it did not.
But at least you can confirm that with manually opening the app, signing was always possible.

But at least you can confirm that with manually opening the app, signing was always possible.

yes so far that's the case in this example.

I tested on iOS Safari.
Generally I can't properly switch between wallets. The main menu to select wallets looks as follows:

The "Browser Wallet" entry just shows the following page (the explanation is confusing – I don't have a browser wallet installed, so it is to be expected that none is found here):

The "All Wallets" entry leads to a long list of browser, however, 1) it does not contain Phantom and 2) it is not able to determine that I have Rabby wallet installed. This is unexpected as the "Browser Wallet" logo in the main menu indicates that both wallets should be well supported.
Metamask

popup to change app always appear
switch to app and display of signing request always happens
after signing it works most of the time (I see something like approvalHash <hash> on the console), but sometimes I instead see this error message:

when I reject the transaction in the wallet, the following messages shows in the webapp log:

Trust Wallet

popup to change app always appear
switch to app and display of signing request always happens
after signing it works most of the time (I see something like approvalHash <hash> on the console), but sometimes I instead see this error message (however, some time after this message appears it seems that the approvalHash <hash> log message is posted as well):

when I reject the transaction in the wallet, the following messages shows in the webapp log:

Zerion Wallet

popup to change app frequently doesn't appear, although the transaction occurs in the wallet to sign when switching to it
when singing, the behavior is similar to the other wallets above

I wanted to test Phantom and Rabby wallet but couldn't for the reasons at the beginning of my comment.
The selected address at the top of the screen overlaps with the logo on mobile:

In general: I think this works better than rainbowkit, but I would like to test Rabby and Phantom as well, in order to see whether the issues with Zerion are an exception.
We should test the proper flow next in order to see whether switching to the anchor UI causes some extra issues.

@TorstenStueber it's possible to define wallets that need to show up in the list. I already did this for Metamask here as it was also missing in the list for some reason. I'll try to do the same for Rabby and Phantom so you can test with them as well.
I'll also remove the custom logic for early signing so it can be tested with the 'normal' flow.

@TorstenStueber you can give this another try. I added the config for rabby and phantom and restored the handler for the 'Confirm' button.

I think that's because neither Rabby nor Phantom appear as 'WalletConnect certified' on the list here.

So the walletconnect support with Appkit seems to be limited to these wallets. Which can be good or bad. Assuming that the 'WalletConnect certified' means that the support is tested, it would ensure that we only show options that are confirmed to work.

The code is ready for review.

did anyone also test on Desktop? Anything not working there?

I missed that message before. I tested this on desktop as well and it works well for me. Tested both the walletconnect option connecting to the desktop browser with the phone (using Zerion) and the usual connection with Metamask and transaction signing was requested as expected.
@pendulum-chain/product do you also want to check out the deploy preview? As the underlying library changed, the connector dialog now looks different. Please have a look and let us know if it's good to go from your side.

Tested on desktop Windows + Chrome.
Got second signing request.

For some reason, the deploy preview failed when building the first time but clearing the cache and retrying a second time worked, see here. So I'll merge this regardless.


chore: temporary fossa fix

Description
Temporary fossa fix to missing dependencies.

:tada: This PR is included in version 1.11.0-beta.10 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Features/dispatch plots

This PR integrates the dispatch plots developed in oemoflex into the processing pipeline. I tested it for FlexMex2_1a
Can you test this for FlexMex2_2a,b,c,d as well, @monika-o? Probably you will have to adapt the plot_labels.yaml and plot_colors.csv.

I've adjusted the labels and colors as well as the carrier name and tested the plots for FlexMex2_1a to d and FlexMex2_2a to d. They all worked. But we also need to update the plots function in oemoflex to add the different electricity demands together.

Thanks for updating, @monika-o! I tested it and got a lot of nice plots. Above are some suggestions for improvement.
Also, I get this warning. Do you get the same. If so, can you find out how to fix that?
Exception in Tkinter callback
Traceback (most recent call last):
  File "/usr/lib/python3.6/tkinter/__init__.py", line 1705, in __call__
    return self.func(*args)
  File "/usr/lib/python3.6/tkinter/__init__.py", line 749, in callit
    func(*args)
  File "python3.6/site-packages/matplotlib/backends/_backend_tk.py", line 253, in idle_draw
    self.draw()
  File "python3.6/site-packages/matplotlib/backends/backend_tkagg.py", line 9, in draw
    super(FigureCanvasTkAgg, self).draw()
  File "python3.6/site-packages/matplotlib/backends/backend_agg.py", line 407, in draw
    self.figure.draw(self.renderer)
  File "python3.6/site-packages/matplotlib/backend_bases.py", line 1544, in _draw
    def _draw(renderer): raise Done(renderer)
matplotlib.backend_bases._get_renderer.<locals>.Done: <matplotlib.backends.backend_agg.RendererAgg object at 0x7f6fdc5a5ef0>

The issue with ktinker is an issue in matplotlib version 3.3.4, which is the version we use and also the latest that I can install with pip. According to https://github.com/matplotlib/matplotlib/issues/20204, the issue is now fixed, at least in the pre-release of version 3.5.0.

The issue with ktinker is an issue in matplotlib version 3.3.4, which is the version we use and also the latest that I can install with pip. According to matplotlib/matplotlib#20204, the issue is now fixed, at least in the pre-release of version 3.5.0.

Ok, let us ignore the warning for now - as soon as the new version of matplotlib is released, it should not appear any more.

I have realized your suggestions. You can now check it once again if you like.

@jnnr I am now done again. I have also tested it with the new processed data and it works both for FlexMex2_1 and FlexMex2_2 (I have tested the first expansion step for both).

The flake8 checks are failing again because the comparison with True was reset to == True in dev. This is necessary because the comparison with numpy.bool would return False otherwise. Can you please tell flake8 to ignore the specific lines, @monika-o?

I just noticed that the units seem to be wrong. What is declared as MW in the dispatch plots should be GW. Please check that and correct it, @monika-o.

@jnnr Unfortunately, I still couldn't solve the problem with flake8. Locally, flake8 accepted the noqa command behind the closing bracket and ignored the error, but apparently this does not work here on github. This is despite the version of flake8 being the same on my computer and here (4.0.1). According to this issue , it could be that noqa only works when it is in the same line as the code that produces the error (which would cause a conflict with black). But I could not test this because changing it, as you can see above, caused conflicts with the base branch.


Issue/cdm 53

What this PR does:
Which issue(s) this PR fixes:
Fixes #
Checklist:

[ ] Automated Tests added/updated
[ ] Documentation added/updated
[ ] CLA Signed:  DataStax CLA

https://github.com/datastax/cassandra-data-migrator/pull/153


[DO NOT MERGE]

Relevant information

Checklist

[ ] I've read the guidelines and understand the acceptance criteria.
[ ] The value of the attribute marked as identifier will be unique and valid.
[ ] I've confirmed that my entity type wasn't already defined. If it is I'm providing an explanation above.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.2 out of 3 committers have signed the CLA.:white_check_mark: durgaprasad95:white_check_mark: manuelrodriguez-nr:x: dsankaraganti-nrYou have signed the CLA already but the status is still pending? Let us recheck it.


optimize: support rpc timout can be customized.

[ ] I have registered the PR changes.

Ⅰ. Describe what this PR did
Ⅱ. Does this pull request fix one issue?

#4190
Ⅲ. Why don't you add test cases (unit test/integration test)?
Ⅳ. Describe how to verify it
Ⅴ. Special notes for reviews

Codecov Report

Merging #4191 (df6972a) into develop (a3f8dce) will decrease coverage by 0.10%.
The diff coverage is 40.00%.

@@              Coverage Diff              @@
##             develop    #4191      +/-   ##
=============================================
- Coverage      49.37%   49.27%   -0.11%
  Complexity      3763     3763
=============================================
  Files            700      700
  Lines          23664    23668       +4
  Branches        2925     2893      -32
=============================================
- Hits           11685    11662      -23
- Misses         10780    10783       +3
- Partials        1199     1223      +24

Impacted Files
Coverage Δ

...ava/io/seata/core/constants/ConfigurationKeys.java
0.00% <ø> (ø)

.../autoconfigure/properties/TransportProperties.java
43.24% <25.00%> (-2.22%)
:arrow_down:

...ava/io/seata/core/rpc/netty/NettyClientConfig.java
44.61% <100.00%> (+0.86%)
:arrow_up:

...obuf/convertor/BranchRegisterRequestConvertor.java
90.47% <0.00%> (-9.53%)
:arrow_down:

...otobuf/convertor/GlobalBeginResponseConvertor.java
92.59% <0.00%> (-7.41%)
:arrow_down:

...otobuf/convertor/GlobalCommitRequestConvertor.java
93.75% <0.00%> (-6.25%)
:arrow_down:

...otobuf/convertor/GlobalStatusRequestConvertor.java
93.75% <0.00%> (-6.25%)
:arrow_down:

...obuf/convertor/GlobalRollbackRequestConvertor.java
93.75% <0.00%> (-6.25%)
:arrow_down:

...protobuf/convertor/RegisterTMRequestConvertor.java
94.73% <0.00%> (-5.27%)
:arrow_down:

...protobuf/convertor/RegisterRMRequestConvertor.java
90.47% <0.00%> (-4.77%)
:arrow_down:

... and 16 more

LGTM 请登记pr信息至changes文件夹1.5.0.md

done.


sticker.js gini

[UnhandledPromiseRejection: This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason "Maksimal 10 detik!".] {
code: 'ERR_UNHANDLED_REJECTION'
} unhandledRejection

cek kembali


Scrape Failed. Caught exception during request

Hi,
I am running this on a brand new Raspberry pi 4 8GB. I am randomly getting response like this:
E2020-12-14 05:44:33,770 [mzn_2] scrape failed
I2020-12-14 05:44:46,499 [mzn_3] not in stock
I2020-12-14 05:44:56,268 [mzn_4] not in stock
E2020-12-14 05:45:01,436 [bstby_1] caught exception during request: HTTPSConnectionPool(host='www.bestbuy.com', port=443): Read timed out. (read timeout=5)
E2020-12-14 05:45:01,437 [bstby_1] scrape failed
E2020-12-14 05:45:06,575 [bstby_2] caught exception during request: HTTPSConnectionPool(host='www.bestbuy.com', port=443): Read timed out. (read timeout=5)
E2020-12-14 05:45:06,575 [bstby_2] scrape failed
E2020-12-14 05:45:36,778 [mzn_2] caught exception during request: Message: invalid session id
E2020-12-14 07:13:20,149 [mzn_2] caught exception during request: Message: unknown error: session deleted because of page crash
from unknown error: cannot determine loading status
from tab crashed
(Session info: headless chrome=83.0.4103.116)
E2020-12-14 07:13:20,150 [mzn_2] scrape failed
I tried running it on my macbook pro 16 and there's no problem.

Im having the same issue with bestbuy.....they probably added something to prevent scrapers on their page.
E2020-12-14 17:32:45,364 [bstby_1] caught exception during request: HTTPSConnectionPool(host='www.bestbuy.com', port=443): Read timed out. (read timeout=5)
E2020-12-14 17:32:45,365 [bstby_1] scrape failed
E2020-12-14 17:32:50,443 [bstby_2] caught exception during request: HTTPSConnectionPool(host='www.bestbuy.com', port=443): Read timed out. (read timeout=5)
E2020-12-14 17:32:50,443 [bstby_2] scrape failed
E2020-12-14 17:32:55,511 [bstby_3] caught exception during request: HTTPSConnectionPool(host='www.bestbuy.com', port=443): Read timed out. (read timeout=5)
E2020-12-14 17:32:55,512 [bstby_3] scrape failed
E2020-12-14 17:33:00,585 [bstby_4] caught exception during request: HTTPSConnectionPool(host='www.bestbuy.com', port=443): Read timed out. (read timeout=5)
E2020-12-14 17:33:00,585 [bstby_4] scrape failed
E2020-12-14 17:33:05,657 [bstby_5] caught exception during request: HTTPSConnectionPool(host='www.bestbuy.com', port=443): Read timed out. (read timeout=5)
E2020-12-14 17:33:05,658 [bstby_5] scrape failed
E2020-12-14 17:33:10,729 [bstby_6] caught exception during request: HTTPSConnectionPool(host='www.bestbuy.com', port=443): Read timed out. (read timeout=5)
E2020-12-14 17:33:10,729 [bstby_6] scrape failed
E2020-12-14 17:33:15,802 [bstby_7] caught exception during request: HTTPSConnectionPool(host='www.bestbuy.com', port=443): Read timed out. (read timeout=5)
E2020-12-14 17:33:15,803 [bstby_7] scrape failed
E2020-12-14 17:33:20,868 [bstby_8] caught exception during request: HTTPSConnectionPool(host='www.bestbuy.com', port=443): Read timed out. (read timeout=5)
E2020-12-14 17:33:20,868 [bstby_8] scrape failed
E2020-12-14 17:33:25,934 [bstby_9] caught exception during request: HTTPSConnectionPool(host='www.bestbuy.com', port=443): Read timed out. (read timeout=5)
E2020-12-14 17:33:25,934 [bstby_9] scrape failed
E2020-12-14 17:33:30,999 [bstby_10] caught exception during request: HTTPSConnectionPool(host='www.bestbuy.com', port=443): Read timed out. (read timeout=5)
E2020-12-14 17:33:31,000 [bstby_10] scrape failed
E2020-12-14 17:33:36,074 [bstby_11] caught exception during request: HTTPSConnectionPool(host='www.bestbuy.com', port=443): Read timed out. (read timeout=5)
E2020-12-14 17:33:36,074 [bstby_11] scrape failed
E2020-12-14 17:33:41,145 [bstby_12] caught exception during request: HTTPSConnectionPool(host='www.bestbuy.com', port=443): Read timed out. (read timeout=5)
E2020-12-14 17:33:41,145 [bstby_12] scrape failed
E2020-12-14 17:33:46,223 [bstby_13] caught exception during request: HTTPSConnectionPool(host='www.bestbuy.com', port=443): Read timed out. (read timeout=5)
E2020-12-14 17:33:46,223 [bstby_13] scrape failed
E2020-12-14 17:33:51,304 [bstby_14] caught exception during request: HTTPSConnectionPool(host='www.bestbuy.com', port=443): Read timed out. (read timeout=5)
E2020-12-14 17:33:51,305 [bstby_14] scrape failed
E2020-12-14 17:33:56,372 [bstby_15] caught exception during request: HTTPSConnectionPool(host='www.bestbuy.com', port=443): Read timed out. (read timeout=5)
E2020-12-14 17:33:56,372 [bstby_15] scrape failed
E2020-12-14 17:34:01,454 [bstby_16] caught exception during request: HTTPSConnectionPool(host='www.bestbuy.com', port=443): Read timed out. (read timeout=5)
E2020-12-14 17:34:01,454 [bstby_16] scrape failed
E2020-12-14 17:34:06,532 [bstby_17] caught exception during request: HTTPSConnectionPool(host='www.bestbuy.com', port=443): Read timed out. (read timeout=5)
E2020-12-14 17:34:06,532 [bstby_17] scrape failed
E2020-12-14 17:34:11,609 [bstby_18] caught exception during request: HTTPSConnectionPool(host='www.bestbuy.com', port=443): Read timed out. (read timeout=5)
E2020-12-14 17:34:11,610 [bstby_18] scrape failed

I increased the minimum timeout interval from 5 seconds to 15 seconds in this commit: https://github.com/EricJMarti/inventory-hunter/commit/1fa0cbacf01f81d90f41875407fb20c9fb427f70
Can you try again using the latest image?
A lot of factors can cause requests to time out (a slow or congested internet connection, using Wi-Fi instead of ethernet, other processes competing for CPU and network resources, the list goes on). I think 15 seconds should be a reasonable timeout to download a webpage, but let me know if you still see timeout issues and I can raise it more.


issues with mbart models

❓ Questions and Help
Thanks for releasing the mbart models! However, we are unable to produce the EN-RO fine-tuned BLEU scores reported in the paper. We get a BLEU score of 26.9, using sacreBLEU's default tokenization, v13. This is well below the 38.5 reported in the README and even below scores reported for WMT16. Here is a complete script to reproduce this; is there anything obvious we are doing wrong?
We have also tried to work with scoring the main, pretrained-only model, and were surprised to find that the names of the parameters seem to change between the main model and fine-tuned one. Perhaps documenting this is beyond the scope of your intentions with releasing the model, but it is a bit confusing when working with these models.
Code
Here is the code we run:
# constants
langs=ar_AR,cs_CZ,de_DE,en_XX,es_XX,et_EE,fi_FI,fr_XX,gu_IN,hi_IN,it_IT,ja_XX,kk_KZ,ko_KR,lt_LT,lv_LV,my_MM,ne_NP,nl_XX,ro_RO,ru_RU,si_LK,tr_TR,vi_VN,zh_CN
MODELDIR=MBART_finetuned_enro
DICT=$MODELDIR/dict.txt
export FAIRSEQ=~/code/fairseq
export PYTHONPATH=$FAIRSEQ
# end constants

SRC=en_XX
TRG=ro_RO

mkdir tmp
sacrebleu -t wmt16 -l en-ro --echo src | spm_encode --model $MODELDIR/sentence.bpe.model > tmp/data.spm.$SRC
sacrebleu -t wmt16 -l en-ro --echo ref | spm_encode --model $MODELDIR/sentence.bpe.model > tmp/data.spm.$TRG

python3 $FAIRSEQ/preprocess.py \
  --source-lang $SRC \
  --target-lang $TRG \
  --testpref tmp/data.spm  \
  --destdir tmp \
  --thresholdtgt 0 \
  --thresholdsrc 0 \
  --srcdict ${DICT} \
  --tgtdict ${DICT} \
  --workers 70

python3 $FAIRSEQ/generate.py $tmpdir \
  --path $MODELDIR/model.pt \
  --task translation_from_pretrained_bart \
  --gen-subset test \
  --max-tokens 1000 \
  -s $SRC \
  -t $TRG \
  --max-sentences 32 \
  --langs $langs > out.wmt19.ro

grep ^H out.wmt19.ro | sort -V | cut -f3 | spm_decode --model $MODELDIR/sentence.bpe.model | perl -pe 's/\[ro_RO\]//' | sacrebleu -t wmt16 -l en-ro -b

What's your environment?

fairseq Version (e.g., 1.0 or master): latest github
PyTorch Version (e.g., 1.0): 1.4.0
OS (e.g., Linux): CentOS 7.5
How you installed fairseq (pip, source): source
Build command you used (if compiling from source): pip install --editable . (within a conda env)
Python version: 3.7.5
CUDA/cuDNN version: 10.1 / 7.6.3
GPU models and configuration: Titan RTX
Any other relevant information:

@mjpost  Please use the tokenizer in the readme. Our output is not BPEed so you need to use the tokenizer to parse the data and then tokenize it.

Thanks for the response. The README says "set tokenizer  here". I presume this means to apply the remove-diacritics.py and normalise-romanian.py scripts in that repo. I will also guess that I should run the Moses tokenizer with the -l ro flag. Doing so:
cat out.debpe | ~/code/wmt16-scripts/preprocess/normalise-romanian.py | ~/code/wmt16-scripts/preprocess/remove-diacritics.py | ~/code/mosesdecoder/scripts/tokenizer/tokenizer.perl -l ro > out.tok
sacrebleu -t wmt16 -l en-ro --echo ref | ~/code/wmt16-scripts/preprocess/normalise-romanian.py | ~/code/wmt16-scripts/preprocess/remove-diacritics.py | /code/mosesdecoder/scripts/tokenizer/tokenizer.perl -l ro > ref.tok
sacrebleu -tok none -s non -b

gives 37.1, which is at least much closer to what's reported in the README.

@mjpost  Very close now. still miss three steps before normalization.
see below
lg=ro
$REPLACE_UNICODE_PUNCT | $NORM_PUNC -l $lg | $REM_NON_PRINT_CHAR | $NORMALIZE_ROMANIAN | $REMOVE_DIACRITICS | $TOKENIZER -no-escape -threads $N_THREADS -l $lg

@mjpost Please let me know if you can reproduce the number that I can close this issue.

I didn't expect that this preprocessing would make such a different.
Running your exact command gives me 37.8:
#!/bin/bash

set -eu

REPLACE_UNICODE_PUNCT=$HOME/code/mosesdecoder/scripts/tokenizer/replace-unicode-punctuation.perl
NORM_PUNC=$HOME/code/mosesdecoder/scripts/tokenizer/normalize-punctuation.perl
REM_NON_PRINT_CHAR=$HOME/code/mosesdecoder/scripts/tokenizer/remove-non-printing-char.perl
REMOVE_DIACRITICS=$HOME/code/wmt16-scripts/preprocess/remove-diacritics.py
NORMALIZE_ROMANIAN=$HOME/code/wmt16-scripts/preprocess/normalise-romanian.py
TOKENIZER=$HOME/code/mosesdecoder/scripts/tokenizer/tokenizer.perl

sys=$1
ref=$2

lang=ro
for file in $sys $ref; do
  cat $file \
  | $REPLACE_UNICODE_PUNCT \
  | $NORM_PUNC -l $lang \
  | $REM_NON_PRINT_CHAR \
  | $NORMALIZE_ROMANIAN \
  | $REMOVE_DIACRITICS \
  | $TOKENIZER -no-escape -l $lang \
  > $(basename $file).tok
done

cat $(basename $sys).tok | sacrebleu -tok none -s none -b $(basename $ref).tok

I run this as:
# ./eval-enro.sh out.clean.wmt19.ro.txt wmt19.en-ro.ro.txt
37.8

with the attached files (the first one being the output).
wmt19.en-ro.ro.txt
out.clean.wmt19.ro.txt

@mjpost This is very helpful!
I am having trouble following what the final 37.8 solution ended up being.
Is out.clean.wmt19.ro.txt generated from your original script or an intermediate result?
Thanks!

I believe it was generated from the fairseq model (so cat fairseq.out | grep -V H^ | cut -f3 | spm_decode).
I never reached their reported score but it was close.

Thanks! Did you ever get to the bottom of the difference between cc25 and en_ro models?

No—we had thought it would be an easy comparison as a baseline, but I couldn't figure it out after putting some time into it. They didn't respond to the point in my second paragraph above.

This seems not quite resolved, particularly: "the names of the parameters seem to change between the main model and fine-tuned one".
@yinhanliu, can you share any insight on why the weights change between cc25 and en-ro?

There are details of the error I ran into and how to reproduce it in #1754.

https://github.com/pytorch/fairseq/blob/18831f9f8353e7b7902f4d9a651463f50f40ce3f/fairseq/models/bart/model.py#L248
this needs to be True to solve #1754
Sorry, I dont quite understand what you tried to do and what failed?

https://github.com/pytorch/fairseq/blob/18831f9f8353e7b7902f4d9a651463f50f40ce3f/fairseq/models/bart/model.py#L248
this needs to be True to solve #1754
Sorry, I dont quite understand what you tried to do and what failed?
You tried to run generation on a pre-trained model? the pre-trained model is a de-noising model, it will copy src to tgt, it never learned translation.

Yes, I understand it's just a de-noiser. But I should at least be able to run it. I gave an EN-DE example in #1754, but if I switch to EN-EN, it still fails.
How do I tell the model that args.layernorm_embedding is true? This isn't a command-line argument but appears to be an internal API model creation parameter. Why is this not stored in the model config itself?
(A similar source of confusion comes from having to pass the list of language codes, instead of just adding these to the model dictionary.)

https://github.com/pytorch/fairseq/blob/7c0ab23d14882d77ae5017ee71085925c5c03373/fairseq/models/transformer.py#L161
you just passing the arg should be fine.  the fine-tune command in Readme gives better info of how to use pre-trained mbart. The generate command is designed for translation model (fine-tuned) only.

That is an argument for training. I am not trying to fine-tune the pretrained model (that works); instead, I am trying to use the pretrained model with fairseq-generate, which doesn't have a --layernorm-embedding flag.

The generate command is designed for translation model (fine-tuned) only.

This was my original question. In principle there is no reason that I should not be able to decode with the pretrained model. My original question was to ask what parameters changed between the pretrained and fine-tuned models (i.e., what prevents the pretrained model from being used in decoding)?

Here is an example where I am trying to use the pre-trained model (not fine-tuned) to find the auto-encoder score of an English input using `fairseq-generate. I can get it to work with the "translation" task. However, I am not sure that I have the correct results.
First, I have to clean the cc25_pretrain model. After rereading the paper, it seems the extra parameters are likely those mentioned in the "Architecture" paragraph in Section 2 of the paper:

We also include an additional layer-normalization layer on top of both the encoder and the decoder, which we found stabilized training at FP16 precision.

It is likely that the fine-tuning is just dropping these. It is easy to remove them with the following script.
import torch
d = torch.load("cc25_pretrain/model.pt")
for extra in ["encoder.layernorm_embedding.bias", "decoder.layernorm_embedding.weight", "decoder.layernorm_embedding.bias"]:
    if extra in d["model"]:
        dell d["model"][extra]
torch.save(d, "cc25_pretrain/model.pt")

Now, the model will work with the "translation" task with fairseq-generate.
Suppose I would like to use the model to find the auto-encoder score for an English sentence. It seems one has to append the language codes to the start and end, like this:
S-8     ▁Other s ▁have ▁dis miss ed ▁him ▁as ▁a ▁joke . [en_XX]
T-8     [en_XX] ▁Other s ▁have ▁dis miss ed ▁him ▁as ▁a ▁joke .
H-8     -29.304781542170865     [en_XX] ▁Other s ▁have ▁dis miss ed ▁him ▁as ▁a ▁joke .
P-8     -45.5193 -36.3480 -17.6285 -33.7401 -18.9918 -41.1282 -45.5779 -44.3245 -14.6398 -14.6273 -33.6461 -22.1626 -12.6281

The probabilities here are very low. The vocabulary is quite large, but I would have expected the decoder prediction of the first token [en_XX] to be very high. Removing the language codes entirely produces much higher sentence-level scores:
S-8     ▁Other s ▁have ▁dis miss ed ▁him ▁as ▁a ▁joke .
T-8     ▁Other s ▁have ▁dis miss ed ▁him ▁as ▁a ▁joke .
H-8     -13.532117797144407     ▁Other s ▁have ▁dis miss ed ▁him ▁as ▁a ▁joke .
P-8     -30.6400 -2.8578 -18.5653 -10.2856 -16.6289 -16.3278 -17.9757 -5.5656 -3.2346 -20.1281 -9.9263 -10.2497Z

But it is hard to know what the right call is here.
Perhaps the language codes are used as the actual <bos> and <eos> tokens? In which case one would have to adapt the translation_from_pretrained_bart task. I have played around with this quite a bit (one has to take care, since that task does not permit auto-encoder scoring out of the box, but assumes you are using two distinct languages). I can do this work, but it would be very helpful to have some technical guidance here on exactly what the model expects. @yinhanliu, perhaps this would be an easy question for you to answer?

Looking at translation_from_pretrained_bart, it seems that adding the source LID token to the end of the source sentence is correct, but that what I need is to use the target-side language code as the decoder BOS:
eos = None
if append_source_id:
    src_dataset = AppendTokenDataset(src_dataset, src_dict.index('[{}]'.format(src)))
    if tgt_dataset is not None:
        tgt_dataset = AppendTokenDataset(tgt_dataset, tgt_dict.index('[{}]'.format(tgt)))
    eos = tgt_dict.index('[{}]'.format(tgt))

I presume that appending here causes the target-side LID token to serve as EOS, and that I have to set it to BOS, too, so that the decoder context is properly initialized.

Hi @mjpost
I'm trying to reproduce the work you have done to find the auto-encoder score.
I deleted the layernnorm embeddings layers:
import torch
d = torch.load("bart/cc25_pretrain/mbart.cc25/model.pt")
for extra in ["encoder.layernorm_embedding.weight", "encoder.layernorm_embedding.bias", "decoder.layernorm_embedding.weight", "decoder.layernorm_embedding.bias"]:
    if extra in d["model"]:
        del d["model"][extra]
torch.save(d, "bart/cc25_pretrain/mbart.cc25/model_no_layernorm_embedding.pt")

Then I updated the dictionary:
for code in [ar_AR] [cs_CZ] [de_DE] [en_XX] [es_XX] [et_EE] [fi_FI] [fr_XX] [gu_IN] [hi_IN] [it_IT] [ja_XX] [kk_KZ] [ko_KR] [lt_LT] [lv_LV] [my_MM] [ne_NP] [nl_XX] [ro_RO] [ru_RU] [si_LK] [tr_TR] [vi_VN] [zh_CN] "<mask>"; do
    echo "$code 1" >> cc25_pretrain/dict.txt
done

Then I pre-processed some english sentences:
fairseq-preprocess --trainpref sample_text --srcdict dict.txt --tgtdict dict.txt --destdir data-sample --source-lang source --target-lang target

Finally I translated using the pre-trained model:
fairseq-generate data-sample  --path $model  --task translation --gen-subset train -t target -s source --bpe 'sentencepiece' --sentencepiece-vocab ../sentence.bpe.model

However I am getting very weird output, for example:
S-0	How are you.[en_XX]
T-0	[en_XX] How are you.
 But I believe the problem is that MBART models use the target-language token both as BOS and EOS for the decoder. That is, the decoder is initialized with the language tag as BOS, and terminates when it generates that tag. This creates the following set of problems:

You cannot actually set BOS. You can prefix decode to a token (i.e., force it to predict the target language token as the first target-side token), but not set it to BOS.

Things are complicated because fairseq uses the values of -s and -t both as indices into the prepacked data and, optionally, to initialize BOS and EOS. But if you are trying to paraphrase, as your example suggests, you have to use a dummy language code for the source sentence, since you need en_XX for the target.

I am also uncertain about using the unadapted model in this way. They train with an additional layernorm layer (which you've deleted above). But without adapting the model, it's not clear what effect removing this will have, since the pretraining made use of it.

In short, I think you have to modify the code to be able to use the model in this way, and it's not certain that, once the technical barriers are cleared, the model will do what you are hoping. But of course, that is what I was trying to test empirically (and the reason for this issue).

@mjpost @yinhanliu @sshleifer I understand you guys trying to replicate the score of the paper but the paper is not comparable to the WMT6 Sennrich's results for EN-RO.
37.8 is computed on tokenized output and reference, and on top of this, there is some normalization  / preprocessing on the reference which is not what they do for WMT.
In the mBart paper it says for RO "We apply Moses tokenization and special normalization for Romanian texts" following Sennrich 2016a but this is wrong to me.
Bottom line @mjpost your 26.9 is the WMT comparable score except if you prefer to normalize / remove diacritics in your output, but not in the reference.

@vince62s Hi, for EN-RO, we were comparing with previous works where people compute tokenized BLEU on the normalized datasets. Thanks

do you mind quoting these "previous works" ? thanks.

@vince62s For example, the unsupervised MT results in MASS (https://arxiv.org/pdf/1905.02450.pdf), XLM (https://arxiv.org/pdf/1901.07291.pdf);
many works about non-autoregressive MT (e.g. https://arxiv.org/pdf/1909.02480v3.pdf, https://arxiv.org/pdf/1904.09324.pdf, ...)

@vince62s we are just following (Sennrich 2016a)'s scripts to produce the normalized/tokenized datasets. https://github.com/rsennrich/wmt16-scripts
I don't know how they evaluated at WMT16 official completion, however, many papers followed their scripts to process the data and computed tokenized BLEU scores.
If you wanted me to dig all the related papers, it will take some time. Thanks

@MultiPath I don't want to sound too critical because the mBart paper is great.
However, the EN-RO statement is wrong.
Just an example, all your reported WMT17/WMT18/WMT19 scores for other language pairs are by far below the state-of-the-art (transformer) or such, just look at WMT official scores.
For EN<>RO I ran Google translate is gives RO>EN 43.3, EN>RO 32.7
So how can your EN>RO be legit ? you would be by far state-of-art on all other language pairs as well.
When you say you just used Sennrich's scripts, nobody contests this, it's just that you don't have the "right" to normalize/touch the reference. You can do whatever you want on the training data, the output of your system, but in the end, you need to score a detokenized output and submit it to sacrebleu or a scorer that uses the Nist/13a/14 tokenization to be comparable, using the reference "as is".
@mjpost who is well aware about sacrebleu may confirm.

in addition, for those who might be interested in the original paper of Rico: https://www.aclweb.org/anthology/W16-2323.pdf
it says clearly:
"We found that the use of diacritics was inconsistent in the Romanian training (and development) data, so for Romanian→English we removed diacritics from the Romanian source side, obtaining
improvements of 1.3–1.4 BLEU.
Synthetic training data gives improvements of 4.1–5.1 BLEU. for English→Romanian, we found
that the best single system outperformed the ensemble of the last 4 checkpoints on dev, and we
thus submitted the best single system as primary system."
=> no diacritics removal in the EN > RO experiment. You can check the output of Rico here:
http://matrix.statmt.org/matrix/output/1843?run_id=4303

@mjpost @myleott
When I reproduced the results, I met some problems. I hope you can give me some suggestions.
I try to finetune on pretrained mbart.cc25 with my own data. But I got a 6.9G model, while the downloaded en2ro model is 2.3G. The only difference between the two models is my finetune model has one more parameter: decoder.output_projection.weight.
I'm not sure why the pretrained model does not have this parameter, but the finetune model does?
I use the master code of fairseq.
`
data_dir=data/j2k_base
model_dir=model/j2k_base
seed=456
max_tokens=1024
PRETRAIN=./mBART/mbart.cc25/model.pt
langs=ar_AR,cs_CZ,de_DE,en_XX,es_XX,et_EE,fi_FI,fr_XX,gu_IN,hi_IN,it_IT,ja_XX,kk_KZ,ko_KR,lt_LT,lv_LV,my_MM,ne_NP,nl_XX,ro_RO,ru_RU,si_LK,tr_TR,vi_VN,zh_CN
src_lan=ja_XX
tgt_lan=ko_KR
python train.py $data_dir
--restore-file $PRETRAIN
--save-dir $model_dir
--langs $langs
--encoder-normalize-before
--decoder-normalize-before
--arch mbart_large
--task translation_from_pretrained_bart
--source-lang $src_lan
--target-lang $tgt_lan
--criterion label_smoothed_cross_entropy
--label-smoothing 0.2
--dataset-impl mmap
--optimizer adam
--adam-eps 1e-06
--adam-betas '(0.9, 0.98)'
--lr-scheduler polynomial_decay
--lr 3e-05
--min-lr -1
--warmup-updates 2500
--total-num-update 40000
--dropout 0.3
--attention-dropout 0.1
--weight-decay 0.0
--max-tokens $max_tokens
--save-interval 10
--update-freq 8
--me 100
--save-interval-updates 100
--keep-interval-updates 100
--no-epoch-checkpoints
--seed $seed
--log-format simple
--log-interval 10
--reset-optimizer
--reset-meters
--reset-dataloader
--reset-lr-scheduler
--layernorm-embedding
--ddp-backend no_c10d
`

Here is an example where I am trying to use the pre-trained model (not fine-tuned) to find the auto-encoder score of an English input using `fairseq-generate. I can get it to work with the "translation" task. However, I am not sure that I have the correct results.
First, I have to clean the cc25_pretrain model. After rereading the paper, it seems the extra parameters are likely those mentioned in the "Architecture" paragraph in Section 2 of the paper:

We also include an additional layer-normalization layer on top of both the encoder and the decoder, which we found stabilized training at FP16 precision.

It is likely that the fine-tuning is just dropping these. It is easy to remove them with the following script.
import torch
d = torch.load("cc25_pretrain/model.pt")
for extra in ["encoder.layernorm_embedding.bias", "decoder.layernorm_embedding.weight", "decoder.layernorm_embedding.bias"]:
    if extra in d["model"]:
        dell d["model"][extra]
torch.save(d, "cc25_pretrain/model.pt")

Next, I manually add the language codes to the dictionary:
for code in [ar_AR] [cs_CZ] [de_DE] [en_XX] [es_XX] [et_EE] [fi_FI] [fr_XX] [gu_IN] [hi_IN] [it_IT] [ja_XX] [kk_KZ] [ko_KR] [lt_LT] [lv_LV] [my_MM] [ne_NP] [nl_XX] [ro_RO] [ru_RU] [si_LK] [tr_TR] [vi_VN] [zh_CN] "<mask>"; do
    echo "$code 1" >> cc25_pretrain/dict.txt
done

Now, the model will work with the "translation" task with fairseq-generate.
Suppose I would like to use the model to find the auto-encoder score for an English sentence. It seems one has to append the language codes to the start and end, like this:
S-8     ▁Other s ▁have ▁dis miss ed ▁him ▁as ▁a ▁joke . [en_XX]
T-8     [en_XX] ▁Other s ▁have ▁dis miss ed ▁him ▁as ▁a ▁joke .
H-8     -29.304781542170865     [en_XX] ▁Other s ▁have ▁dis miss ed ▁him ▁as ▁a ▁joke .
P-8     -45.5193 -36.3480 -17.6285 -33.7401 -18.9918 -41.1282 -45.5779 -44.3245 -14.6398 -14.6273 -33.6461 -22.1626 -12.6281

The probabilities here are very low. The vocabulary is quite large, but I would have expected the decoder prediction of the first token [en_XX] to be very high. Removing the language codes entirely produces much higher sentence-level scores:
S-8     ▁Other s ▁have ▁dis miss ed ▁him ▁as ▁a ▁joke .
T-8     ▁Other s ▁have ▁dis miss ed ▁him ▁as ▁a ▁joke .
H-8     -13.532117797144407     ▁Other s ▁have ▁dis miss ed ▁him ▁as ▁a ▁joke .
P-8     -30.6400 -2.8578 -18.5653 -10.2856 -16.6289 -16.3278 -17.9757 -5.5656 -3.2346 -20.1281 -9.9263 -10.2497Z

But it is hard to know what the right call is here.
Perhaps the language codes are used as the actual <bos> and <eos> tokens? In which case one would have to adapt the translation_from_pretrained_bart task. I have played around with this quite a bit (one has to take care, since that task does not permit auto-encoder scoring out of the box, but assumes you are using two distinct languages). I can do this work, but it would be very helpful to have some technical guidance here on exactly what the model expects. @yinhanliu, perhaps this would be an easy question for you to answer?

I think the fact that the probability of the langid token is low might be related to this. I don't know how the model was trained, but based on the implementation of multilingual_denoising on master, the  langid token is not used as the BOS in the input to the decoder.

I didn't expect that this preprocessing would make such a different.
Running your exact command gives me 37.8:
#!/bin/bash

set -eu

REPLACE_UNICODE_PUNCT=$HOME/code/mosesdecoder/scripts/tokenizer/replace-unicode-punctuation.perl
NORM_PUNC=$HOME/code/mosesdecoder/scripts/tokenizer/normalize-punctuation.perl
REM_NON_PRINT_CHAR=$HOME/code/mosesdecoder/scripts/tokenizer/remove-non-printing-char.perl
REMOVE_DIACRITICS=$HOME/code/wmt16-scripts/preprocess/remove-diacritics.py
NORMALIZE_ROMANIAN=$HOME/code/wmt16-scripts/preprocess/normalise-romanian.py
TOKENIZER=$HOME/code/mosesdecoder/scripts/tokenizer/tokenizer.perl

sys=$1
ref=$2

lang=ro
for file in $sys $ref; do
  cat $file \
  | $REPLACE_UNICODE_PUNCT \
  | $NORM_PUNC -l $lang \
  | $REM_NON_PRINT_CHAR \
  | $NORMALIZE_ROMANIAN \
  | $REMOVE_DIACRITICS \
  | $TOKENIZER -no-escape -l $lang \
  > $(basename $file).tok
done

cat $(basename $sys).tok | sacrebleu -tok none -s none -b $(basename $ref).tok

I run this as:
# ./eval-enro.sh out.clean.wmt19.ro.txt wmt19.en-ro.ro.txt
37.8

with the attached files (the first one being the output).
wmt19.en-ro.ro.txt
out.clean.wmt19.ro.txt

Hi Matt,
Thank you for your script which really helps me a lot.
Following what you did, I can reproduce a BLEU score of 37.5, which is still 1 BLEU score lower than the paper and 0.3 lower than your result.
I have three questions about the issue:

Should I also normalize the testing input (i.e., English text) with:
| $REPLACE_UNICODE_PUNCT
| $NORM_PUNC -l $lang
| $REM_NON_PRINT_CHAR
| $TOKENIZER -no-escape -l $lang \

$(basename $file).tok

Should I truecase the test sets?

Have you successfully reproduced the result of 38.5 BLEU score? if so, could you share the key tricks?

Thanks,
Xuebo Liu

Hi @SunbowLiu, I don't know the answers to your questions and gave up on trying to exactly recreate their scores.

Hi @vince62s,
I totally agree with your concerns about the tokenization of the RO reference and also somewhat doubt about the real fine-tuning performance.
But after finetuning the RO-EN model by myself over mbart.cc with the WMT16 training data, I can successfully reproduce a sarceBLEU score of 37.5 with the command cat test.hyp | sacrebleu -t wmt16 -l ro-en. Noted that test.hyp is detokenized text which makes the evaluation fully the same to the official WMT. The result indeed outperforms previous SOTA, confirming the effectiveness of mbart.

@SunbowLiu I am also trying to finetune with torch 1.5.1, apex, fairseq on master and getting this issue  https://github.com/NVIDIA/apex/issues/161 . What versions are you running?

@SunbowLiu my point on diacritics is on EN to RO (not the reverse).
Of course you can beat some WMT16 results with RO to EN while at that time it was RNN time ....

Hi @vince62s,
I totally agree with your concerns about the tokenization of the RO reference and also somewhat doubt about the real fine-tuning performance.
But after finetuning the RO-EN model by myself over mbart.cc with the WMT16 training data, I can successfully reproduce a sarceBLEU score of 37.5 with the command cat test.hyp | sacrebleu -t wmt16 -l ro-en. Noted that test.hyp is detokenized text which makes the evaluation fully the same to the official WMT. The result indeed outperforms previous SOTA, confirming the effectiveness of mbart.

@SunbowLiu  Do you mean that 37.5 is the sarceBLEU of WMT16 RO-EN? How about the sarceBLEU of WMT16 RO-EN with back-translation?

@moussaKam I haven't had time to look into this deeply enough to solve it, so this may be incorrect. But I believe the problem is that MBART models use the target-language token both as BOS and EOS for the decoder. That is, the decoder is initialized with the language tag as BOS, and terminates when it generates that tag. This creates the following set of problems:

You cannot actually set BOS. You can prefix decode to a token (i.e., force it to predict the target language token as the first target-side token), but not set it to BOS.
Things are complicated because fairseq uses the values of -s and -t both as indices into the prepacked data and, optionally, to initialize BOS and EOS. But if you are trying to paraphrase, as your example suggests, you have to use a dummy language code for the source sentence, since you need en_XX for the target.
I am also uncertain about using the unadapted model in this way. They train with an additional layernorm layer (which you've deleted above). But without adapting the model, it's not clear what effect removing this will have, since the pretraining made use of it.

In short, I think you have to modify the code to be able to use the model in this way, and it's not certain that, once the technical barriers are cleared, the model will do what you are hoping. But of course, that is what I was trying to test empirically (and the reason for this issue).

Hi Matt, I've tried to feed the target [en_XX] as the first input token of decoder, However, this could not make some differences.  The pre-trained mbart even gets extremely low probability  (lower than uniform distribution) yielded by force-decoding. Do you have any ideas about this? Than you.

Hi @vince62s,
I totally agree with your concerns about the tokenization of the RO reference and also somewhat doubt about the real fine-tuning performance.
But after finetuning the RO-EN model by myself over mbart.cc with the WMT16 training data, I can successfully reproduce a sarceBLEU score of 37.5 with the command cat test.hyp | sacrebleu -t wmt16 -l ro-en. Noted that test.hyp is detokenized text which makes the evaluation fully the same to the official WMT. The result indeed outperforms previous SOTA, confirming the effectiveness of mbart.

Could you share the fine-tuning script? Do you have results on fine-tuning mbart for EN-RO?


Experiment with storing output timestamps in .tsbuildinfo

Occasionally, TypeScript makes the choice not to write output files in --watch mode if the output is the same as what's on disk; however, this causes complications with --incremental. Under --incremental, you --watch still needs to update each output file's timestamps so that a subsequent invocation of tsc --build will avoid rebuilding certain files. See more here: https://github.com/microsoft/TypeScript/commit/7b290fdbd4e892379c3ab3ed46d7b41f15a31a2b.
One concern with this is that it causes issues with downstream tooling - messing with output files that wouldn't have been rewritten causes issues for tools that want to be more incremental as well. See https://github.com/microsoft/TypeScript/issues/46661.
Another concern is that this causes a bunch of file-touching anyway which can potentially be slow. It's not clear whether this is really a bottleneck, but there is some discussion in https://github.com/microsoft/TypeScript/issues/45082 around issues here, and it may warrant some investigation.

Note: A potential further consideration may be around browser caching of the output files. If the modified time of the file changes then browsers may not cache the unchanged file. Just trying to add weight to this really painful issue (for us).

@rbuckton try tackling this sooner in the 4.6 cycle, loop @jakebailey, @amcasey, and me in reviews.


Cannot kill long running process

There is no indicator how long a running process will take to finish and no way to stop a running processes without closing brackets.

This is logged as an enhancement, not a bug.  Otherwise, things are working as designed.


banned

Just got banned in owo for an hour
owo recognizes the macro and bans the acc for an hour
any way to fix this? by tweaking the time of something? please inform me!

#111
As per my knowledge it's better to increase time.


dotnetCorePackages.net_5_0 is out of date, and out of sync with sdk_5_0

Checklist

[x] Checked the nixpkgs master branch

[x] Checked the nixpkgs pull requests

#112582 is also out of date

Project name
nix search name: (I'm not sure how to check this on a system with flakes)

current version: 5.0.0 / 5.0.3
desired version: 5.0.6
Additional note
I have a PR open for packaging netcoredbg (#103940). Building netcoredbg requires knowing the version of the runtime bundled with the SDK. Currently, the net_5_0 package and sdk_5_0 package are out of sync, making this difficult. The easiest solution would be to make sure to keep these in sync, but another option might be to add something like passthrough.runtimeVersion.
Notify maintainers

maintainers: @kuznero
Note for maintainers
Please tag this issue in your PR.

Fixed by https://github.com/NixOS/nixpkgs/pull/138296
What's a bit confusing is that release 5.0.10 includes two SDKs (see https://dotnetcli.blob.core.windows.net/dotnet/checksums/5.0.10-sha.txt
):

dotnet-sdk-5.0.207
dotnet-sdk-5.0.401

I do not understand the difference nor why there are two inside the same release.
I have upgraded the dotnet 5 sdk to the latter in nixpkgs.


dont render false values

Hyperdom renders false values as the string false which means we are forced to use
this._somethingIsDefined ? renderSomething() : undefined

This PR ignores the false values and allows this._somethingIsDefined && renderSomething() like react

Hey @dobiedad, do you have any email to contact you at?


Check that version number in "Add/Remove programs" matches manifest version during manifest validation

Description of the new feature/enhancement
As reported here, the winget upgrade command may incorrectly detect updates to be available even when the latest version is installed in case the version number in Add/Remove programs does not match the version in the manifest.
I suggest that the manifest validator checks for this condition automatically after program installation and either fails the validation process (requiring the manifest submitter to adapt the manifest version) or creates a warning so that the submitter can take action if they want.
This suggestion is closely related to #13622.

I also suggest that winget validater should use winget update to test submitted installers with older versions (like vs code), then use winget uninstall to test if it can be uninstalled normally.

Is there any chance that the test cases are opened up at some point, at the very least so we can run them locally? The community could write some of this code, especially for the smaller stuff (product code bug, for instance).

@jedieaston we do want to open validation up. We have a bunch of infrastructure work to do this. We've been discussing some different architectural approaches to make this possible. We got a lot of mileage out of Azure DevOps, but we're starting to see some other approaches that might scale better, and give the community more surface area to help.
I can't express in text "here" how much we all appreciate the support and constructive feedback!

@denelon Looks like this one was missed with Area-Validation-Pipelines


Can't search for dollar sign, even when escaped.

I tried searching my project for the angularJS service $timeout using ag. I tried the following:

$ ag "$timeout"

However, it acts like there were no results. In grep if I type this:

$ grep -rl "$timeout" ./

It will find them all. Naturally I'd prefer to use my lightning fast silver searcher over grep ;)
Any idea why I can't escape the dollar sign? I think it's still using it as the end of line regular expression even when escaped.

Is there by any chance a switch to automatically escape dollar signs et al. when using a password file via -f?

Is there by any chance a way to automatically escape dollar signs et al. when reading passwords from a file using -f?


Execute the taskDidComplete closure in addition to the default implementation

We ran into the problem that when the taskDidComplete closure was set the response handlers were never executed because the closure replaced the default implementation.
This change fixed the problem for us and we think it would be useful for everyone.

Thank you for your PR @oanhof.
The taskDidComplete closure is exposed to allow you to completely override the default behavior if you so choose. It was not intended to allow you to hook into the SessionDelegate callbacks for notifications and still execute the default behavior.
As an alternative, you can use the response serializers to know once a task was completed. Additionally, we plan to open up more of the Request class in the 1.3.0 release to allow you to extend a Request to have additional hooks along with the already provided ones such as validate, authenticate, response, responseJSON, etc.
Hopefully this helps clarify things a bit. Cheers.


Chat - Sent message is hidden for the Group Chat when we use the GSD mode

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Expected Result:
Sent message is visible for the Group Chat when we use the GSD mode
Actual Result:
Sent message is hidden for the Group Chat when we use the GSD mode
Action Performed:

Open app and login with Account A
Go to the Setting and click on Preferences
Use Priority Mode - GSD
In another browser (or incognito window) log in as account B
Click on + click Create a Group
Select 4-5 users and include Account A.
Send messages to the Group

Workaround:
Unknown
Platform:
Where is this issue confirmed?
Web
mWeb ✔
iOS
Android
Desktop App
Mobile Web
Version Number: 1.0.12-0
Logs: https://stackoverflow.com/c/expensify/questions/4856
Notes/Photos/Videos:
Video

Expensify/Expensify Issue URL:

@roryabraham not sure if we should add the deploy blocker label in this one. The issue is not reproducible in production but these seems like intended changes https://github.com/Expensify/Expensify.cash/pull/2015 made here.

I actually think this is intended behavior, so I think we should close this out.


Pin Pandas version

The latest version of pandas doesn't have pandas.tseries.offsets.BusinessMixin

Maybe it's just moved to https://github.com/pandas-dev/pandas/blob/bfac13628394ac7317bb94833319bd6bf87603af/pandas/_libs/tslibs/offsets.pyx#L1204
Given that it's only a pre-release, what's the best approach here? Pin pandas pre v1.1 or have some hack to look in the right place depending on which pandas it is?

Or do we even need the BusinessMixin support? We don't care about resampling onto business days.
A simple try: except ImportError block should work fix this

Or do we even need the BusinessMixin support? We don't care about resampling onto business days.

yep can easily drop


Classes annotated with @ConfigPrefix can define the prefix so you dont need to in the getConfig() call.

Addresses issue https://github.com/gestalt-config/gestalt/issues/58


fix wrong file naming

The logging module is currently writing the diff-log as diff-log.txt file, while restore and diff are using diff-log.csv naming convention.
Either:

change the name used by diff and restore modules to be diff-log.txt
change the name used by the logging module to be diff-log.csv

Closed by #102


build: build/gen everything out of src dir

We don't want to have anything being built or generated inside the src
dir, this patch removes all the remaining resources being generated
there.
The only remaining resource being generated in the source dir is the
oic.json - which in my option should remain there.
See: https://github.com/solettaproject/soletta/issues/561
Signed-off-by: Leandro Dorileo<EMAIL_ADDRESS>

v2 sent...


Improve documentation for Beta release

Motivation
For beta release it would be good to improve documentation to make it more clear what is the purpose of Zebra and how to install and run it.
Specifications

Designs

Solution

Closes https://github.com/ZcashFoundation/zebra/issues/2870
Review
Just a draft for now. Feel free to suggest any changes.
Not very confident on the content, just wanted to get it started.
Something I'm not sure about: if we choose to avoid the repetition between README and user guide about setup procedure, should we keep it in the README or in the user guide?

Reviewer Checklist

[ ] Code implements Specs and Designs
[ ] Tests for Expected Behaviour
[ ] Tests for Errors

Follow Up Work

Something I'm not sure about: if we choose to avoid the repetition between README and user guide about setup procedure, should we keep it in the README or in the user guide?

I have no strong opinions here.
But in general, I think a shorter README is better, as long as it has a brief summary of the setup.

Great start! Thank you for getting the conversation going on this :)


Removed EF Core 2.1 note

EF Core 2.1 is out and already supports data seeding, so this note is no longer relevant.

@KirkMunro Another nice improvement to the doc. Thank you again!


x/website: /VERSION endpoint returns outdated information

What version of Go are you using (go version)?
N/A
Does this issue reproduce with the latest release?
N/A
What operating system and processor architecture are you using (go env)?
N/A
What did you do?
$ curl 'https://go.dev/VERSION?m=text'
go1.17.4

What did you expect to see?
go1.17.5 since the current release per go.dev/dl is 1.17.5
What did you see instead?
go1.17.4
Adding my request trace just in case it helps debugging this.
x-cloud-trace-context: ee059a46e3120e5832ea0c9571613f8c
date: Sun, 12 Dec 2021 14:47:46 GMT
server: Google Frontend
content-length: 8

I can confirm the behaviour described here.

Duplicate of #42744


fixing scan_onsets issue in brsa

Make sure that run_TRs calculated from scan_onsets is integer. And make scan_onsets integers in the example.

@mihaic Hi, can you approve this minor change? Thanks!

@lcnature You have a conflict with the master branch. You can either fix the conflict yourself or accept the PR I made for your branch.

@I accepted it.

@mihaic I accepted it.