MUPX: nginx config?

Sounds like MUPX is configuring NGINX in a dockr container.   I want to deploy multiple virtual hosts.  How do I setup the nginx reverse proxy within dockr?

+1

+1

meteorhacks:cluster says you don't need nginx?  Your thoughts please.
https://github.com/meteorhacks/cluster

Not for clusters, but the use case I am describing is to support multiple
environments on the same server (e.g. demo, qa, dev).  I want to use nginx
virtual servers with a single external IP.  This is not problem if I
manually configure nginx using regular MUP.  With MUPX the nginx is running
in a Dockr container.  I'm not quite sure how to configure a single
"front-end" instance of nginx in this environment.
On Fri, Sep 25, 2015 at 11:36 AM, Master James<EMAIL_ADDRESS>wrote:

meteorhacks:cluster says you don't need nginx? Your thoughts please.
https://github.com/meteorhacks/cluster
—
Reply to this email directly or view it on GitHub
https://github.com/arunoda/meteor-up/issues/661#issuecomment-143184063.

Maybe this is what you're looking for? It's for SSL support
https://github.com/meteorhacks/mup-frontend-server

[Noted: It calls for meteorhacks:mup-ssl-server in the run.sh but I can find it public.]
Anyway it installs it via the Dockfile call to here...
https://github.com/meteorhacks/mup-frontend-server/blob/master/lib/install-nginx.sh
I'm thinking you'd have to modify something in the configuration,
https://www.linode.com/docs/websites/nginx/how-to-configure-nginx
but I didn't really understand your use case fully yet, is there a good keyword for the method you seek like virtual servers?
http://stackoverflow.com/questions/13788146/how-can-meteor-handle-multiple-virtual-hosts
Oh wait a minute does this help with various other resources for nginx configuration?
https://github.com/meteorhacks/mup-frontend-server/blob/master/lib/nginx.conf

I see you mentioned Reverse Proxy initially, so I guess this kind of Reverse Proxy stuff...
https://www.nginx.com/resources/admin-guide/reverse-proxy/
goes into your local copy of
https://github.com/meteorhacks/mup-frontend-server/blob/master/lib/install-nginx.sh

@lookingcloudy I need to do the same.
I have multiple meteor apps running on the same server. I want different subdomains to access different apps.
I was planning on using nginx to do that. Did you have any luck?

Having the same issue. I'm trying to do some subdomain routing with nginx (e.g. point http://blog.myapp.com at something else) and was wondering how to do this.

I have done this successfully by manually installing and configuring nginx.

Deploy meteor normally using mupx with these environment variables in mup.json:
 "env": {
     "BIND_IP": "<IP_ADDRESS>",
     "PORT": 3000,
     "ROOT_URL": "http://a.example.com"
 },

The port can be anything except 80. The BIND_IP variable tells meteor only to listen to requests originating from the machine itself. This way some other computer can't connect directly to meteor at http://a.example.com:3000 but nginx can connect to meteor1.

Install nginx on the server however you like. Its probably as easy as sudo apt-get install nginx.

Find out where your nginx config files live and where you are supposed to add new sites2. Add a new site for your meteor server:
server {
    listen 80;
    server_name a.example.com;

    location / {
        proxy_pass http://<IP_ADDRESS>:3000;

        proxy_http_version 1.1;

        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
    }
}

This tells nginx that any requests for the host a.example.com on port 80 for any path should be passed on to http://<IP_ADDRESS>:3000 which is your meteor app. The proxy header settings are for websockets and stuff. I'm not sure on the details but I got help from this meteorpedia article and other places around the net.

Repeat for any other meteor apps, making sure that each meteor server gets a different port (e.g. 3001) and a different server name (e.g. b.example.com).

I actually couldn't get mupx to successfully deploy with this variable set, for unrelated reasons. Luckily, the setup will still work without it.
This is the standard way to configure nginx to serve sites, so do your own searching to find out how nginx works on your server.

+1

@BudgieInWA currently about to do this for a deployment I have on AWS. Thanks for the post very useful

I used force-ssl package and everything is working sweet on EC2 AWS

Has anyone been able to set this up with a custom nginx and multiple meteor apps with ssl? Im currently working on it and Ive been able to get 2 apps on a server using different ports but as soon as I try to add ssl everything break, I get either 502 bad gateway or connection refused. Any help?

@ivanthemeh, one way to do it is to use nginx for the SSL termination. This means that communication between the web browser and nginx is encrypted and the communication between nginx and your meteor server is not (which is fine, because they're both on the same machine). Using the approach that I described above, only the nginx config needs to be changed. The new config might look something like this:
server {
    listen                443 ssl;
    server_name a.example.com;

    ssl_certificate       /etc/letsencrypt/live/a.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/a.example.com/privkey.pem;

    location / {
        proxy_pass http://<IP_ADDRESS>:3000;

        proxy_http_version 1.1;

        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
    }
}

Setting up SSL is out of scope for this thread, but it's standard practice so there are plenty of resources out there.

@BudgieInWA Hey man thanks for this Im going to start working on it now, If I need more help Ill hit you up again. Thanks so much!

@BudgieInWA Hey man I just wanna say thanks alot! With your help and a bit of researching I was able to succesfully get two meteor apps running on one ec2 instance with ssl for each!! 👍 Thanks soo much man!


Error every time you type while 'resources' is expanded in the JSON Outline

Repro Steps:

Open an ARM template
Expand the 'resources' node in the JSON Outline view
Start typing in the resources section of your ARM template

Expected: No errors
Actual:
Pop this error every time I type a character:

I specifically repro-ed this when I was trying to add a new resource. So hit enter after an existing resource, enter the curly brace, and it should happen. I guess that might be the only time this happens


bugfix: do not display peer review marks in course grades summary

Proposed Changes
(Describe your changes here. Also describe the motivation for your changes: what problem do they solve, or how do they improve the application or codebase? If this pull request fixes an open issue, use a keyword to link this pull request to the issue.)
Currently in the grade summary table, peer review grades are displayed in the column for the parent assignment, with actual grades on the parent assignment not being viewable. This PR fixes that bug by removing peer review data from the gradde summary table.
...

Screenshots of your changes (if applicable)
Before:
![image](https://github.com/MarkUsProject/Markus/assets/3107290/8accf71c-e6a9-401d-9dca-181ed36d10c6)
After:

Associated [documentation repository](https://github.com/MarkUsProject/Wiki) pull request (if applicable)

Type of Change
(Write an X or a brief description next to the type or types that best describe your changes.)

Type
Applies?

🚨 Breaking change (fix or feature that would cause existing functionality to change)

✨ New feature (non-breaking change that adds functionality)

🐛 Bug fix (non-breaking change that fixes an issue)
x

🎨 User interface change (change to user interface; provide screenshots)

♻ Refactoring (internal change to codebase, without changing functionality)

🚦 Test update (change that only adds or modifies tests)

📦 Dependency update (change that updates a dependency)

🔧 Internal (change that only affects developers or continuous integration)

Checklist
(Complete each of the following items for your pull request. Indicate that you have completed an item by changing the [ ] into a [x] in the raw text, or by clicking on the checkbox in the rendered description on GitHub.)
Before opening your pull request:

[x] I have performed a self-review of my changes.

Check that all changed files included in this pull request are intentional changes.
Check that all changes are relevant to the purpose of this pull request, as described above.

[x] I have added tests for my changes, if applicable.

This is required for all bug fixes and new features.

[ ] I have updated the project documentation, if applicable.

This is required for new features.

[ ] If this is my first contribution, I have added myself to the list of contributors.

After opening your pull request:

[x] I have updated the project Changelog (this is required for all changes).
[x] I have verified that the pre-commit.ci checks have passed.
[x] I have verified that the CI tests have passed.
[x] I have reviewed the test coverage changes reported by Coveralls.
[ ] I have requested a review from a project maintainer.

Questions and Comments
(Include any questions or comments you have regarding your changes.)
I had to disable the Rails/PluckId cop on two lines in my tests, because it warns to use .ids instead of .pluck(:id) even when plucking from an array of hashes and not an activerecord collection.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

26 of 26 (100.0%)  changed or added relevant lines in 3 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.005%) to 91.391%

Totals

Change from base Build<PHONE_NUMBER>:
0.005%

Covered Lines:
40076

Relevant Lines:
43171

💛  - Coveralls

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

26 of 26 (100.0%)  changed or added relevant lines in 3 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.004%) to 91.396%

Totals

Change from base Build<PHONE_NUMBER>:
0.004%

Covered Lines:
40103

Relevant Lines:
43198

💛  - Coveralls


hash oai identifier for ES identifier

Logs are showing some OAI identifiers failing as id's, consider md5 hashing them.
Update: Imagine this means that OAI identifiers acquired through harvesting, which are then used as record_id in Combine, some fail due to character encoding.
Have not seen this in quite awhile now, but keeping issue open, as it's still not a bad idea.  It would be a shame to lose that semantic meaning from the record_id column, but worth considering.

Fixed with combine_id.


How to save Postgres Array type in v4

actually, i have done the same, but i have a problem, How can i save Postgres Array type in v4
because in v3 it was in Column typ but i dono how can i do it.
can you help me please ?

Hello @kenanshrbage.
Currently, the Strapi team is working on full support for custom fields, so far I will hold off on updating this plugin.
You can track updates there: https://feedback.strapi.io/developer-experience/p/custom-fields

FYI, work on the new version is in progress :)

Strapi v4 is supported now by default


Title Bar Customization Setting Conflicts with Context Menu Style

Hey, I've just customized my code editor with this extension, and I'm satisfied. However, when I set some settings for the Title Bar like this:
"apc.electron": {
  "titleBarStyle": "hidden"
},
"window.titleBarStyle": "native",
"window.customTitleBarVisibility": "never"

It hides the title bar, which is what I want. Unfortunately, this also sets the Context Menu style to Windows native style, which is not what I want. I prefer the default context menu style.

Is there any way I can fix this in Windows 10? Here's the VS Code version:
Version: 1.86.2 (user setup)
Commit: 903b1e9d8990623e3d7da1df3d33db3e42d80eda
Date: 2024-02-13T19:40:56.878Z
Electron: 27.2.3
ElectronBuildId: 26908389
Chromium: 118.0.5993.159
Node.js: 18.17.1
V8: <IP_ADDRESS>-electron.0
OS: Windows_NT x64 10.0.19045

Thank You, have a great day!

"window.titleBarStyle": "native",

Thank you. That makes sense. One last question, can I hide this project title and file name from the title bar and keep the rest visible?

This will make my customization 100% completed!

  "apc.stylesheet": {
    ".window-title": {
      "display": "none"
    }
  }

Thanks a zillion @drcika! I owe you one ⚡


Handlebars inside

Description
When handlebars tags are included inside

This would be a subset of #660.  HTML beautifier does some support for handlebars, but js beautifier does not.  The code inside the script tag is processed using the js beautifier.

Is this same issue?
<script>
  let a = {
    b: async function () {

      },
      c: function () {

      }
  }

</script>

This can be controlled by the templating: 'handlebars' setting added in 1.10.0


Heroku suspended the AdoBot-IO app.

The same problem


Flash effects from outside a container can still effect people inside a container

Description

Stuff yourself in a locker, have a flash effect go off outside, still receive the flash vision effect on your vision
Reproduction

Screenshots

Additional context

This is probably a general problem with InRangeUnOccluded then, which is used by the FlashArea function. You would expect an entity to be occluded inside a container.


Windows 11 IoT (Enterprise) GpioController support

Trying to use .NET/UWP GpioController for GPIO on Latte Panda Mu with Full Carrier Board on Windows 11 IoT Enterprise (x64), but this appears to fail:
Unhandled exception. System.NotSupportedException: No GPIO controllers exist on this system.
   at System.Device.Gpio.Drivers.Windows10Driver..ctor()
   at System.Device.Gpio.GpioController.GetBestDriverForBoardOnWindows()
   at System.Device.Gpio.GpioController.GetBestDriverForBoard()
   at System.Device.Gpio.GpioController..ctor(PinNumberingScheme numberingScheme)
   at System.Device.Gpio.GpioController..ctor()
   at Program.<Main>$(String[] args) in C:\Users\Niels\source\repos\GpioDemo\Program.cs:line 7
   at Program.<Main>(String[] args)

e.g. with simple code like:
new System.Device.Gpio.GpioController()

I have installed drivers as found here in this github repo and device manager shows the following for system devices. Maybe something is missing or?
Or how can I use the GPIO from .NET 8/9 in Windows?

the GPIO pins are not yet supported but you can use the I2c and the uart stuff.

That is unfortunate, since we solely bought the LattePanda expecting Gpio to work? When is this expected to be working and what are the pre-requisites for this to work?

sadly i'm not sure when they will release a bios with the gpio pins working. Now you can use the I2C and UART. As of right now i'm using the I2c's. You can use a i2c to GPIO expander. I have 12 different sensors and multiple relays hooked up to one i2c port.

I was in the same boat, purchased it for the gpio but i've that the I2c  (though i do need more external hardware) actually exceeds what i need and it works very well.


[Snyk] Security upgrade tap from 11.1.5 to 12.0.2

Snyk has created this PR to fix 1 vulnerabilities in the npm dependencies of this project.
Snyk changed the following file(s):

package.json
package-lock.json

Vulnerabilities that will be fixed with an upgrade:

Issue
Score

Incomplete List of Disallowed Inputs SNYK-JS-BABELTRAVERSE-5962463
  786

[!IMPORTANT]

Check the changes in this PR to ensure they won't cause issues with your project.
Max score is 1000. Note that the real score may have changed since the PR was raised.
This PR was automatically created by Snyk using the credentials of a real user.

Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.
For more information:
🧐 View latest project report
📜 Customise PR templates
🛠 Adjust project settings
📚 Read about Snyk's upgrade logic

Learn how to fix vulnerabilities with free interactive lessons:
🦉 Learn about vulnerability in an interactive lesson of Snyk Learn.

:tada: Snyk checks have passed. No issues have been found so far.
:white_check_mark: security/snyk check is complete. No issues have been found. (View Details)
:white_check_mark: license/snyk check is complete. No issues have been found. (View Details)
:white_check_mark: code/snyk check is complete. No issues have been found. (View Details)


Landing page example is broken

The landing page example readCSV("data//people.csv") is broken from a user-perspective because the user does not have the csv file.
Expected: docs examples should preferably refer to built in data sets. Only examples relating reading data clearly must refer to files (which a users typically will replace by her own ones).

Hi, @holgerbrandl could we say, that for 0.10 release it's not actual for now (there is no people.csv usage on the docs)
But I reported two other issues related to yours
You suggested here a good solution

Thanks for keeping track of my question. Built-in data sets help to spread usage. E.g. most R examples on stackoverflow use built-in data to make answers reproducible for anyone out of the box.

Could I close this for the moment? Yes, we going to solve this during this release cycle

Sure thing, whatever works best to make kotlin-dataframe even better. Looking forward to it!


Decompiler should split stores that write to multiple struct fields

What is the feature you'd like to have?
Suppose the compiler optimizes two stores to adjacent 32-bit values into a single 64-bit store instruction.  I would like Binary Ninja to split it back into multiple stores when applicable based on type information.  (Same applies to loads.)
For example, this example binary initially decompiles as:
00000000  void start(int64_t* arg1)
00000008      *arg1 = 0x4358000043c00000

I happen to know that the 64-bit constant 0x4358000043c00000 actually represents a pair of 32-bit floats, 384.0 and 216.0.  So I create a struct in the Types view:
struct point __packed
{
    float x;
    float y;
};

And I set the type of arg1 to struct point *.
This just changes the decompiled output to:
00000000  void start(struct point* arg1)
00000008      arg1->__offset(0x0).q = 0x4358000043c00000

But I would like something more like
00000000  void start(struct point* arg1)
00000008      arg1->x = 384.0
00000008      arg1->y = 216.0

Is your feature request related to a problem?
It's difficult to make sense of these kinds of writes otherwise.
Are any alternative solutions acceptable?
Some way to mark individual instructions to be split (as opposed to determining it based on types) might also work, and might be more flexible in some cases (e.g. with unions), though it would be less convenient in the common case.

This should have been fixed in builds >= 3563. Please let us know if it's not.


setUserId is not a function

Trying to use this library and keep getting this error, my code looks like this:
chatbase.newMessage()
      .setAsTypeUser()
      .setPlatform(event.address.channelId)
      .setVersion(version)
      .setTimestamp(new Date(event.timestamp).getTime())
      .setMessage(event.text)
      .setUserId(event.address.user.id || event.user.id)
      .send()
      .then(msg => console.log(msg.getCreateResponse()))
      .catch(err => console.error(err));

Error I get is:
TypeError: chatbase.newMessage(...).setAsTypeUser(...).setPlatform(...).setVersion(...).setTimestamp(...).setMessage(...).setUserId is not a function

Running this on:

Node v7.5.0
MacOS Sierra v10.12.6

What am I doing wrong?
Thanks

Hey @beldar thanks for report looking into this now!

@beldar Issue is fixed on master, will release an update to the NPM package and ping this thread.

The fix is now uploaded to the package on NPM:
https://www.npmjs.com/package/@google/chatbase
Version: 0.0.2
Please re-open this issue if this did not fix the issue you were having. Thanks again for reporting this!

It did fix this issue, other ones have sprung up, but I'll open other tickets for them.
Thanks for the quick fix! 👌


0.60.2 Release date?

I just want to know if 0.60.2 is going to be released shortly. Otherwise I'll have to release my clone to NPM as react-datepicker makes yarn fail when imported from github.

Which commit are you looking for? Does 0.61.0 fixes the issue for you?

Oh sorry I thought the next version was 0.60.2.
No 0.61.0 doesn't fix it, it's the very latest commit I'm looking for: f49251167b6f0fb9cb356736cf4dbac0247856f1

We'll get a new release out this week. Most likely today or tomorrow.

Thanks.


Implementation of a SimpleBuildWrapper for use in workflows. Change i…

…n parent version due to "Build wrappers" headline here: https://github.com/jenkinsci/workflow-plugin/blob/cd9b86e225c772084c39ebb110275bf076f010ad/COMPATIBILITY.md

This is great. I would merge with tests, please!

I read about that but noticed there was no test for you build wrapper so I thought I might get away with it ;) But I'll produce some test and update the PR.

Well now you also have to write a test for the build wrapper :)

Can we get this merged? I am itching to have this support as I convert projects to Workflow. I love AnsiColor!

Sorry, I didn't get a notification from Github from the update - thanks for writing the tests. I think some questions from @jglick seem legit, care to take a look please @qvicksilver. If you don't mind squashing commits too.

Merging.

Retroactively filed as JENKINS-30528.


Corsair K100 Keyboard (Lighting Zone Pictures Included)

Please add support for the following device:
Device Name: Corsair K100 RGB Optical-Mechanical Gaming Keyboard
Device Website: https://www.corsair.com/ww/en/Categories/Products/Gaming-Keyboards/RGB-Mechanical-Gaming-Keyboards/K100/p/CH-912A01A-CH

Any progress on this?  Is there a way to edit the device list and clone/modify existing corsair keyboards to support the K100?

I believe the Corsair SDK does not expose the additional lighting zones around the edges. The rest of the keys work correctly with one of the other Corsair keyboard profiles.

Using "Keyboard Audio Visualizer" the dev said that the latest Corsair SDK does expose all LEDs and it works without any fancy setup with the KAV program.  I asked the Dev for that who said the program only uses the Corsair iCue SDK, no other tricks.

I have a k100 and I would love for this to be added. It kind of acts funky with some keys. I love this software a lot and I want to be able to use it, but I'm not going to be able to like it a lot if it doesn't work right. Hope this device gets added soon. :)


Dice Loss vs Dice Loss with smooting term

This issue discusses differences in the implementation of the Dice Loss with and without the smoothing term.
Background why opening this issue/discussion
tl;dr:

nnunetv2 nnUNetTrainerDiceCELoss_noSmooth trainer (i.e., without the smoothing term of the Dice loss) helped the model from collapsing to zero during lesion model training.

Details
Since the default nnUNetTrainer trainer was collapsing to zero when training the DCM (degenerative cervical myelopathy) lesion model, we tried nnUNetTrainerDiceCELoss_noSmooth (i.e., without the smoothing term of the Dice loss).
This trainer was discovered by @naga-karthik in these two nnunet threads (1, 2). The trainer indeed helped, and the model was no longer collapsing to zero; see details in this issue.

Comparison of the default and nnUNetTrainerDiceCELoss_noSmooth trainers
tl;dr:

the default nnUNetTrainer trainer uses smooth: float = 1.
nnUNetTrainerDiceCELoss_noSmooth uses 'smooth': 0

Details
nnunetv2 default trainer
The nnunetv2 default trainer uses MemoryEfficientSoftDiceLoss (see L352-L362 in nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py).
This MemoryEfficientSoftDiceLoss (see L58 in nnunetv2/training/loss/dice.py) uses both smoothing term (self.smooth) and small constant (1e-8); see L116:
dc = (2 * intersect + self.smooth) / (torch.clip(sum_gt + sum_pred + self.smooth, 1e-8))

nnunetv2 nnUNetTrainerDiceCELoss_noSmooth trainer
The nnunetv2 nnUNetTrainerDiceCELoss_noSmooth trainer (see L32 in nnunetv2/training/nnUNetTrainer/variants/loss/nnUNetTrainerDiceLoss.py) sets smooth to 0. The small constant (1e-8) is apparently untouched and kept.

What is the smoothing term used for?
tl;dr: hard to say convincingly.

keras and ivadomed use only the smoothing term without the small constant
nnunetv2 default trainer uses both the smoothing term and the small constant
nnunetv2  nnUNetTrainerDiceCELoss_noSmooth trainer uses only the small constant (because the smoothing term is set to zero)

Details
Initially, I incorrectly thought that the nnunetv2 smoothing term was used to prevent division by zero. I got this sense based on this comment. But, after a deeper look at the equation in this comment, I found out that the equation uses only the smoothing term but no small constant. Further investigation led me to these two discussions (1, 2) about the Dice implementation in keras. Both
discussions use only the smoothing term but again, but no small constant:
score = (2. * intersection + smooth) / (K.sum(y_true_f) + K.sum(y_pred_f) + smooth)

Checking the ivadomed Dice implementation, and finding that it also uses only the smoothing term (see L63 in ivadomed/losses.py):
return - (2.0 * intersection + self.smooth) / (iflat.sum() + tflat.sum() + self.smooth)

I also found this comment from Charley Gros providing the following explanation (note that this comment is related to the ivadomed Dice without the small constant):

A very probable reason is the different way these two functions are dealing with empty GT and empty pred.
--> Dice loss returns 1

Both keras and ivadomed implementations are in contrast with the nnunet implementation, which uses both smoothing term (self.smooth) and small constant (1e-8); see L116:
dc = (2 * intersect + self.smooth) / (torch.clip(sum_gt + sum_pred + self.smooth, 1e-8))

Prompting chatGPT to explain why removing the smoothing term from the nnunet dice helped from collapsing to zero, provides a relatively reasonable explanation (highlighted in italics):

The smoothing term, as used here, serves a dual purpose:

Stabilization of the Dice Coefficient: By adding the smoothing term (self.smooth) to both the numerator (nominator) and the denominator, it ensures that even when there are no true positives (TP) (e.g., the model predicts no lesion where there is actually no lesion, or it fails to predict a lesion where there is one), the Dice coefficient does not become undefined or result in division by zero. The use of torch.clip with a minimum value of 1e-8 further ensures that the denominator cannot be zero, which prevents division by zero errors.
Influence on Gradient Behavior and Model Sensitivity: The smoothing term affects the gradient of the loss with respect to the model's predictions. When TP is very small or zero, the addition of the smoothing term to both the numerator and denominator ensures that the model still receives a gradient signal that encourages it to improve its predictions. It essentially softens the harshness of the Dice loss, making it more forgiving in the early stages of training or in cases where the model struggles to detect small or sparse lesions. This can lead to a more stable and gradual learning process.

In your specific case, removing the smoothing term with the nnUNetTrainerDiceCELoss_noSmooth trainer likely made the loss more directly reflective of the actual performance of the model in terms of the overlap between the predicted segmentation and the ground truth. For the task of lesion segmentation, where lesions can be small and the balance between lesion and non-lesion areas is crucial, the direct feedback without the smoothing term's moderation might have better aligned the loss with the task's objectives, thereby improving model performance.

Further investigation and experiments comparing the nnunet default nnUNetTrainer trainer and nnUNetTrainerDiceCELoss_noSmooth are in progress.
Tagging @naga-karthik and @plbenveniste, who both also work on lesion segmentation. If any of you had time to go through the investigation above to check if I didn't make any naive mistakes, it would be great.

What do you mean by collapsing to zero? Class imbalance was too high so that the model output zeros everywhere after 1000 epochs?
If so, what is the behavior of the best checkpoints (as opposed to final checkpoint)?

What do you mean by collapsing to zero? Class imbalance was too high so that the model output zeros everywhere after 1000 epochs?

The model was crashing to zero after 100-250 epochs, depending on the fold. See the training progress in this comment.

@valosekj ok. nnunet struggles with your second class (which I'm guessing is the lesion class).
Have you tried opening an issue or discussion on the nnunet repo? Last time I checked I remember the main contributor was still pretty active. He might have some good insights on this phenomenon?

@valosekj ok. nnunet struggles with your second class (which I'm guessing is the lesion class).

Exactly!

Have you tried opening an issue or discussion on the nnunet repo? Last time I checked I remember the main contributor was still pretty active. He might have some good insights on this phenomenon? Because that behavior is a little bit weird

We solved the collapsing to zero by using the nnUNetTrainerDiceCELoss_noSmooth trainer based on these two nnunet threads (1, 2), as I tried to describe in the first comment. If the first comment is unclear, please let me know, and I will rephrase it.
This discussion aims to figure out what smoothing is responsible for and why removing it helped in model training.

Does this only happen with region-based training? We trained a model on very small objects, although the class imbalance was maybe less pronounced than yours

Does this only happen with region-based training? We trained a model on very small objects, although the class imbalance was maybe less pronounced than yours (without collapse)

For the region-based model, the training was collapsing to zero after 100-250 epochs (details here).
For the multi-channel model (trained using T2w_ax image and spinal cord segmentation as input channels to segment lesions as output), the model was learning nothing (details here). Changing the trainer to nnUNetTrainerDiceCELoss_noSmooth for the multi-channel model helped; see here.

Looking into MONAI DiceLoss:
f: torch.Tensor = 1.0 - (2.0 * intersection + self.smooth_nr) / (denominator + self.smooth_dr)

where
smooth_nr: a small constant added to the numerator to avoid zero.
smooth_dr: a small constant added to the denominator to avoid nan.

with default values
smooth_nr: float = 1e-5,
smooth_dr: float = 1e-5,

This indicates that both "smoothing" terms in MONAI implementation are basically just small constants allowing the division. This is in contrast with the "smoothing" term equal to 1 as used in keras, ivadomed, and nnunetv2.

"smoothing" term equal to 1 as used in keras, ivadomed, and nnunetv2.

I think this is the major issue with the dice loss implementations in those packages. Having a big term (i.e. 1) is interfering with the loss calculation (and consequently the gradient signals passed through the network) when learning to segment small objects (i.e. lesions)

very interesting. In this comment, Fabian reports similar problem on the LIDC dataset, which is a lesion segmentation task like yours. From my understanding, the Dice loss can fail in 2 ways:

intersection=0; in this case, we would get a dice loss of 0, regardless of if the GT/pred are empty (in which case we would like to have a value of 1 instead of 0, has Charley mentioned, hence the smoothing term in the numerator)
denominator=0 (or close to it): this would give us a dice loss of NaN.I think we can safely say this is NOT what's going on in your case, because otherwise the progress.png would not display values. Taken from this comment:

NaNs would be recognizable in progress.png because they cannot be plotted. So the lines would stop existing or have holes.

Based on this, we can safely say the problematic part is the intersection (otherwise you would get NaNs). I think the fact that your dataset and the LIDC datasets are problematic is because of this intersection term. Because your masks are mostly empty, this intersection is very close to 0 (remember the dice loss takes a softmax as input - not a binary mask, so the intersection CAN be in [0,1]). The signal is too weak and, as @naga-karthik mentioned, the value 1 overshadow the weak signal you have inside the intersection term.
Maybe something to try would be to hardcode a different smoothness term in the dice computation. I reckon a smaller value would not make the training collapse. If that is the case, we could report it back to the nnunet guys.

Thanks for your thoughts, @hermancollin ! I think we can safely proceed with how MONAI has implemented DiceLoss (i.e. setting smoothing to a small constant such as 1e-5, which should be small enough to work with lesion segmentation problems and others where the object-to-segment is large


Peer-to-peer review

Hi guy👋🏽,  kindly give feedback to improve this project.

Congrats 🎉
Your work is awesome 🥇 🎊 , I like it more when you used a background in every book .. could you please refresh the GitHub page in the live demo inside your readme file so the visitor in your repo will be able to see your project.


fix(self-shutdown): abort frozen ios when unsharing shutdown nexus

These frozen IOs prevent the nexus from shutting down. We don't have any hooks today to do this whilsts the target is stopping so we add a simple loop which tries a number of times. Ideally we should get some sort of callback to trigger this.

The frozen IOs are held up in our NexusChannel and not yet (re)submitted into the child device io channel, because in self shutdown path we haven't brought back io mode to normal. So wondering why they hold up nexus unshare?

The frozen IOs are held up in our NexusChannel and not yet (re)submitted into the child device io channel, because in self shutdown path we haven't brought back io mode to normal. So wondering why they hold up nexus nvmf subsystem unshare/stop? Perhaps since the IOs have entered the nexus subsystem and not ack'ed?

Haven't looked into the subsystem for a while but IIRC it wont stop because there's outstanding IO. When the last IO completes the subsystem is stopped.


Inconsistent date output in Datepicker

Today is today at current time.

Yesterday is yesterday at midnight.
Probably need to feed midnight into the new Date()call or produce the date with the just days and not time. Or we need to switch everything to midnight on output.

I went through the code of DatePicker and what I found is that we can't directly produce the date with the just days and not time as, parsed data is of type [object Date] which by default includes time and we can't/shouldn't define our own string format to represent date so I think shifting to midnight is the best option here. I have started working on a PR for this issue.

@futuremint What do you think about changing dates values (which always include time) in DatePicker to midnight of the user's timezone? Right now, it's inconsistent and is sometimes the current time and sometimes midnight and has been this way for 2+ years.
This will result in a breaking change, and I'm not sure if there is a good way to warn the user about it. One option is that we could rename the prop callback and keep the current inconsistency in the current set of props in case someone is working around this bug.


Clicking picture added into camera example

Added a "click_pic()" function to take the picture and it even saves it in the format as almost all camera applications do in the format of "IMG_26-12-2016_11:30:23". This name tells that it is image file clicked on 26-12-2016 at 11:30:23.

Colin : characters do not work as file names on Windows.
YYYYmmdd-HHMMSS format provides an international date format that sorts both numerically and alphabetically.  The shorter name makes it easier to pick out in small message boxes.

@rjt @dessant Thanks for reviewing, I will correct these mistakes.

Please make sure that travis and appveyor builds succeed, you can click on the icon near the commits to get more details.

@dessant Thank you .... Actually I'm new at this .... That was my first PR ... that's why it took that much time. ☺☺


Add support for contrast-adaptive sharpening in 3D

This is an older, easier to implement variant of CAS as a pure fragment shader. It doesn't support upscaling, but the upscaling part of CAS has been superseded by FSR 1.0 anyway.
The sharpening intensity can be adjusted on a per-Viewport basis. For the root viewport, it can be adjusted in the Project Settings.
This can likely be backported to the GLES3 renderer, but I don't know whether this can work in GLES2.
Edit: Done for GLES3 in https://github.com/godotengine/godot/pull/47416. It can't be implemented in GLES2 due to textureLodOffset() not being available.
Test project: https://github.com/Calinou/test_contrast_adaptive_sharpening
This closes https://github.com/godotengine/godot-proposals/issues/1519.
Preview
FXAA is enabled on both screenshots.
Slider comparison
Sharpening disabled

Sharpening enabled (intensity 1.0)

I have two concerns:

The CAS in the example image is introducing significant darkening that wasn't present in the original image
The test case is totally untextured, but CAS is made to restore contrast in textures that have been blurred by FXAA.

The CAS in the example image is introducing significant darkening that wasn't present in the original image

I think these ringing artifacts are expected when you use a high intensity. Most of the time, you won't be using 1.0 but something between 0.0 and 1.0 (0.4 is a good start).

From a testing standpoint I would pick a scene that is a model of typical fps scene.
The TPS demo isn't typical of a first person pbr view but we have it easily! https://github.com/godotengine/tps-demo

I've made a test project where you can easily toggle FXAA, CAS and camera animation: https://github.com/Calinou/test_contrast_adaptive_sharpening
One issue is that HDR highlights are lost when CAS is enabled. We need to fix this before this can be merged, so I'll mark this PR as a
draft.

Rebased and tested again, it works successfully.

Rebased and tested again, it works successfully.

Rebased and tested again, it works successfully.
Sharpening
Filmic tonemapping (whitepoint = 6)

ACES tonemapping (whitepoint = 0.5)

Zooming in on the image you can see the artifacts that result from CAS the way it is implemented right now. You essentially end up with a black outline on sharp corners.

Rebased and applied the fix from https://github.com/godotengine/godot/pull/51841.

Note: This pull request will most likely not be merged until a decision is made about FSR: https://github.com/godotengine/godot/pull/51679

Rebased and tested again with TAA enabled (and TAA + FXAA), it works as expected.

Might be worth making some visualizations of scenes with TAA with and without CAS.

Might be worth making some visualizations of scenes with TAA with and without CAS.

I made a comprehensive set of comparison images, along with recommended sharpening factors for each antialiasing type:

1024×600: https://imgsli.com/MTEyOTg4
2560×1440: https://imgsli.com/MTEyOTg4

You can do that without an extra pass by storing the intermediate results in local shared memory and then in the CAS filter take your samples from LSM instead of reading from the old texture.

Does this mean I need to create a shared vec3 variable with a size that matches the viewport's width and height?

No, the shared variable should only be 9x9 vec3s.The shared memory is not shared by the entire shader invocation. It is shared on a compute unit. Since we structure our compute shader work in 8x8 thread groups and the CAS filter only reads neighbouring pixels you just need to do the copy to a 9x9 area for each group.
I can send you a nice presentation that explains this in more detail (it also has good images which makes this way easier to understand)

Tagging for 4.x. If this is finished before 4.0 we can merge it, but there is no rush.

@Calinou, I'm tagging this as salvageable since many people still desire the feature.

@Calinou, I'm tagging this as salvageable since many people still desire the feature.

FSR2 has its own CAS implementation, so I don't think this is as important nowadays. The native TAA solution is generally not as good as FSR2 (especially since our TAA is quite slow), so I wouldn't spend too much effort on this.

Terrain3D and similar approaches do not play well with temporal effects like TAA, FSR, and physics interpolation. Until we have a way to neutralize motion vectors or be excluded from these effects, all non-temporal effects should be on the table.

@Calinou, since CAS is a non-temporal effect, I think this would be useful for VR.
Are you interested in updating the pull request to the latest master 4.4?

since CAS is a non-temporal effect, I think this would be useful for VR.

As far as I know, sharpening is discouraged in VR as it'll exaggerate the appearance of aliasing (any aliasing becomes more noticeable).

The reasoning for sharping is that many of the antialiasing techniques for games cause a sort of general blue and you would use CAS to restore sharpness. That was my experience with the unreal engine in 2016; things may have changed.


operators-installer - protect against job names that are to long

What is this PR About?
the verifyer and approver job names can get to long, need to put in smarts to truncate them.
How do we test this?
added a test, but actaully the test doesnt trigger the length problem because i can't find an operator thats name is to long now that i have shortnered the job name to just be "verifer" and "approver" rather then "install-plan-verifer" and "install-plan-approver" but you can run a local helm template with a made up operator name that is long and see it fail.
if anyone happens to know of an operator csv that is 54+ characters let me know.
cc: @redhat-cop/day-in-the-life

first install-test fail was due to some timeout issue or something as the test that failed wasn't even touched by this change. tryig again.


Fix 48 more possible memory confusion bugs

I found 48 more incorrect slice constructions in the native package and fixed them. These are similar to #68.

Coverage decreased (-0.04%) to 73.144% when pulling 35fc7b7caaa4759e2c8c7af5bd737aa74e813fdb on jlauinger:fix-unsafe-slices into 123d3a8a8e2cd3db678ccc397df861e1d64cec18 on gorgonia:master.


Can not add a post description

Before submitting an issue, make sure you have:

[x] Updated to the lastest version v1.4.0
[x] Read the README and docs
[x] Searched the bugtracker for similar issues including closed ones
[x] Reviewed the sample code in tests and examples

Which client are you using?

[x] app (instagram_private_api/)
[ ] web (instagram_web_api/)

Purpose of your issue?

[x] Bug report (encountered problems/errors)
[ ] Feature request (request for a new functionality)
[ ] Question
[ ] Other

Code:
# Example code that will produce the no error reported
from instagram_private_api import Client, ClientCompatPatch

def get_image_for_post(media_path):
	with open(media_path, "rb") as img:
		f = img.read()
		b = bytearray(f)
	return b

user_name = 'user'
password = 'name'

api = Client(user_name, password)
bytear = get_image_for_post('/tmp/1/2018-02-17_00:39:28320.jpg')
api.post_photo(photo_data=bytear, caption="caption_text", size=(1080, 1080))

Describe your issue
Hey. For some time, the function of adding a description to the post has stopped working. Photo is successfully fasted, but the description is still not there. I'm not using api correctly?

@Maxfashko
Your issue report does not conform to the issue template that has been specified for this repo:
https://raw.githubusercontent.com/ping/instagram_private_api/master/.github/ISSUE_TEMPLATE.md
Please edit your issue to comply with the template requirement.
This issue will be closed after 24 hours if no followup action is taken.

[This comment is auto-generated. ref=notemplate]

I just tested this and it worked just fine. You may be shadow banned.

I used several accounts and connected via VPN.

Neither of which does not mean you're not shadow banned.
Your caption can also cause IG to shadow ban posts. Especially if you are spamming hashtags.

How do I get around this? The matter is that I did not specify hashtags. I specified only the text

There is no "get around". You simply have to stop whatever that you were doing that IG doesn't like.


VM is missing jq command

/linux-container-workshop/hol-content/05-kubernetes-ui.md
tells the user to use NAME=$(az group list | jq '.[0]."name"' -r)
But the VM does not have jq installed
sudo yum install jq is one option.. or pre-install it ?

@kevingbb fixed 22 days ago


Bav = 10.0e-44

Dear GoodVibes developers,
I tried to reproduce the entropy contributions in the low frequency limit (Grimme, DOI: 10.1002/chem.201200497) and I discovered that you are using a value of BAV of
10.0e-44 (line 1922 in GoodVibes.py) while Grimme suggested in his paper to use 1e-44.
Was it your intention to change this value or just a typo in the code?
Best regards,
Pascal Friederich

Good spot, thanks! I played around with this value to see its effect and it's possible that it got changed at some point from the value in Grimme's original CEJ paper. In testing, changing by an order of magnitude didn't seem to give rise to noticeable effects - in this case, I have now changed it back to 1e-44, which for our systems leads to a difference of around 2 mH (0.001 kcal/mol) in free energy at 298K.

When we played around with B(av) between 1e-45 to 1e-43 (DOI: 10.1021/acs.joc.8b02436) the variation on the absolute free energies was on the order of 0.1 kcal/mol, although for free energy differences the effect was about 0.01 kcal/mol.

Thank you!


Update Moq

Part of the fix for aspnet/Coherence-Signed#583.

@Eilon already got the OSS approval.


EULA feature integration

would like to know SYM is having built in functionality for the end user license agreement page ? Is that possible to integrate its a wonderful feature to have who are using the SYM by manually enrolling the device
https://github.com/swiftDialog/swiftDialog/issues/328#issuecomment-1885977450

i recently implemented something similar in my own SYM iteration, please see the following as an example:
title="CIO Technology Services" icon="/Library/Application Support/Dialog/Dialog.png" message="**Please Confirm the EULA to Enroll with $$$$$$$$**  \n\nBy clicking \"I Confirm\", you acknowledge that this device is provided by $$$$$$$$$ and is **NOT** a **PERSONAL** or **CLIENT-OWNED** Mac.  \n\nOnly devices provided by $$$$$$$$ are authorized to enroll into $$$$$$$ Systems.  \n\n**It is against $$$$$$$$$ Policy 57 to enroll non-$$$$$$$$$ Mac devices.**" button1="Continue" button2="Cancel" updateScriptLog "Prompting user to confirm that the device is $$$$$$$$ provided" userClick=$("$swiftDialog" --icon "$icon" --title "$title" --message "$message" --eula --checkbox "I Confirm this is an $$$$$$$ provided Mac authorized to enroll into $$$$$$$ Systems",enableButton1 --button1disabled --button1text "$button1" --button2text "$button2" --infobox "**Action Required**  \n\nConfirm the EULA:  \n\nEID: ${loggedInUser}  \nDevice: ${serialNumber}" --height 475 --width 550) userClick=$? if [[ $userClick == 0 ]]; then     updateScriptLog "Clicked confirm.. proceeding to enrollment" elif [[ $userClick == 2 ]]; then     updateScriptLog "User did not confirm device is $$$$$$$$$ provided, removing from Jamf, removing MDM profiles and Jamf framework"     sudo /usr/local/jamf/bin/jamf policy -event deleteComputerRecord     sudo /usr/local/jamf/bin/jamf removemdmprofile     sudo /usr/local/jamf/bin/jamf removeframework     exit 1 fi
the $$$$$ is where i redacted the company name, hope this helps

Thanks for the FR, @mani2care.
After counseling with other sym-devs, I'm going to close this as "not planned."
(However, I think someone might have an idea for you …)


Bug in tests: instruction test doesn't seem to check the witness

The test for a new instruction I am implementing passed, but when I run it manually and check the result with pilcom, the constraints doesn't check out.

It seems riscv heavy tests uses compile_asm_string_temp(), instead of the pilcom verifying alternative verify_asm_string(). I tried using verify_asm_string() in there, but was OOM-killed. Not sure it is worth it, as the the compiler has its own asm tests that do verify with PILCOM, so they are supposed to cover for everything.

Done.


Fix typos

This PR fixes numerous typos detected using codespell.

@microsoft-github-policy-service agree

Fantastic, thanks! It looks like https://github.com/microsoft/PlanetaryComputerExamples/pull/236/files#diff-077a90e70172de3617c1d3eb588b769f7e94bf0377bf81fd70d48fbef9c0ca56 might have been an accidental removal of an output. I'll push a commit restoring that.
I might also add a pre-commit hook to keep these from slipping back in.

I might also add a pre-commit hook to keep these from slipping back in.

That might need to wait on something like https://github.com/codespell-project/codespell/issues/2138, to avoid false positives on "typos" in the image outputs we check in here. So I'll punt on that for now.


HADOOP-18102. Extract method to check two URIs are from same hosts

Description of PR
Extracted the logic out to a new method. Ran existing unit tests. Added a new test
How was this patch tested?
For code changes:

[ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
[ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
12m 19s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
32m 20s

trunk passed

+1 :green_heart:
compile
22m 21s

trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
compile
19m 44s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
checkstyle
1m  8s

trunk passed

+1 :green_heart:
mvnsite
1m 39s

trunk passed

+1 :green_heart:
javadoc
1m 15s

trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
javadoc
1m 42s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
2m 29s

trunk passed

+1 :green_heart:
shadedclient
22m 30s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 58s

the patch passed

+1 :green_heart:
compile
21m 39s

the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
javac
21m 39s

the patch passed

+1 :green_heart:
compile
19m 37s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
javac
19m 37s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

-0 :warning:
checkstyle
1m  7s
/results-checkstyle-hadoop-common-project_hadoop-common.txt
hadoop-common-project/hadoop-common: The patch generated 2 new + 89 unchanged - 0 fixed = 91 total (was 89)

+1 :green_heart:
mvnsite
1m 39s

the patch passed

+1 :green_heart:
javadoc
1m  9s

the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
javadoc
1m 48s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
2m 40s

the patch passed

+1 :green_heart:
shadedclient
22m 36s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
17m 28s

hadoop-common in the patch passed.

+1 :green_heart:
asflicense
0m 58s

The patch does not generate ASF License warnings.

209m 16s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3957/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/3957

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell

uname
Linux 1c9b0f3a105a 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / dbcf6a00ab03b5c147ae80b40374abf084a73700

Default Java
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3957/1/testReport/

Max. process+thread count
1294 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3957/1/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.


Resize Image

Hello,
Thanks for your wonderful code! It is very helpful and I am trying to use it for my Mturk survey.
I would like to resize the image and I have tried to change the width of image_element but it seems that instead of resizing, the image will be cropped.
I would be grateful if you could let me know whether it is possible to resize the image by some changes in the code.

@nargeshn See README

If I just change the width and height of image_element in BBoxAnnotator, the image is cropped as I mentioned in the first comment. So in addition to changing width and height in BBoxAnnotator, in "annotator.image_frame.css" I added "background-size": "500px 500px" (the size that I want) and it worked.

Thanks @nargeshn for the solution. However, do you know how can we change the size based on aspect ratio? For example, we just provide the image height and it resizes the width based on the aspect ratio of original image.

@chahna107 given only desired image height/width, calculate the image width/height based on the original image aspect ratio. Simply replace
options.width || (options.width = image_element.width);
options.height || (options.height = image_element.height);

with the following code
if (options.width) {
    let aspect_ratio = image_element.width / image_element.height;
    options.height = Math.round(options.width / aspect_ratio);
} else if (options.height) {
    let aspect_ratio = image_element.width / image_element.height;
    options.width = Math.round(options.height * aspect_ratio);
}
else {
    options.width = image_element.width;
    options.height = image_element.height;
}

Note this is done in the compiled js code.

"background-size": "500px 500px"

You realy saved my day :)


Separate JUnit code from core

Extract a core module that does not have a dependency on JUnit

Would be cool to have example tests that work with Spock

Seems to be closed premature.

There are a couple of thing remain after introduction of MesosClusterTestRule:

the MesosClusterTestRule should move to a separated module;
minimesos should lose compilation dependency on junit

Fixed after recent refactoring.


Batoto and empty series.

BIG EDIT, my mistake. At first the following error happened with the same series which was empty of chapters, now exchanging elements in my download list the behavior has changed, so I can't say it is related to a particular "status" of the series.
At first with the following list:
http://bato.to/comic/_/comics/3d-kanojo-r8943
http://bato.to/comic/_/comics/12-beast-r8357
http://bato.to/comic/_/comics/14-sai-no-koi-r1970
http://bato.to/comic/_/comics/7-seeds-r402
The error was:
Error: Invalid HTTP response: http://bato.to/comic/_/comics/14-sai-no-koi-r1970
at HttpServiceError (..\mangarack-component-core\src\services\HttpServiceError.ts:13:5)
at Request._callback (..\mangarack-node-core\src\services\httpService.ts:123:16)
at self.callback (../mangarack-node-core/node_modules/request/request.js:198:1)
at emitOne (events.js:77:13)
at Request.emit (events.js:169:7)
at [object Object]._onTimeout (../mangarack-node-core/node_modules/request/request.js:811:1)
at Timer.listOnTimeout (timers.js:89:15)
Then with the following elements in my list:
http://bato.to/comic/_/comics/14-sai-no-koi-r1970
http://bato.to/comic/_/comics/7-seeds-r402
The error happened with the second series:
Fetching 14-sai no Koi
Finished 14-sai no Koi (00:00:07)
Error: Invalid HTTP response: http://bato.to/comic/_/comics/7-seeds-r402
at HttpServiceError (..\mangarack-component-core\src\services\HttpServiceError.ts:13:5)
at Request._callback (..\mangarack-node-core\src\services\httpService.ts:123:16)
at self.callback (../mangarack-node-core/node_modules/request/request.js:198:1)
at emitOne (events.js:77:13)
at Request.emit (events.js:169:7)
at [object Object]._onTimeout (../mangarack-node-core/node_modules/request/request.js:811:1)
at Timer.listOnTimeout (timers.js:89:15)

So.. this happened randomly? I'm a little confused here.

I don't think it is random, but it does not depend on the series, just on the order of the list or something like that.
I'll try to be schematic.
The command is always the same:
call %AppData%\npm\mangarack-cli --component.core.batoto.username XellossMetallium --component.core.batoto.password PASSWORD < Z.txt --runnable.cli.disableNormalize 1
With the list:
http://bato.to/comic/_/comics/zelphy-of-the-aion-r8500
http://bato.to/comic/_/comics/zero-no-tsukaima-chevalier-r1421
http://bato.to/comic/_/comics/zetman-r1089
http://bato.to/comic/_/comics/zetsuen-no-tempest-r465
http://bato.to/comic/_/comics/zettai-shoujo-seiiki-amnesian-r10454
http://bato.to/comic/_/comics/zero-in-r1781
http://bato.to/comic/_/comics/zero-no-tsukaima-r822
http://bato.to/comic/_/comics/zero-no-tsukaima-zeros-summer-holiday-doujinshi-r5269
I have this:
Fetching Zelphy of the Aion
Finished Zelphy of the Aion (00:00:07)
Fetching Zero no Tsukaima Chevalier
Finished Zero no Tsukaima Chevalier (00:00:05)
Error: Invalid HTTP response: http://bato.to/comic/_/comics/zetman-r1089
at HttpServiceError (..\mangarack-component-core\src\services\HttpServiceError.ts:13:5)
(the rest is as above)
But if I switch a couple of entries in the list, like this:
http://bato.to/comic/_/comics/zetman-r1089
http://bato.to/comic/_/comics/zelphy-of-the-aion-r8500
http://bato.to/comic/_/comics/zero-no-tsukaima-chevalier-r1421
http://bato.to/comic/_/comics/zetsuen-no-tempest-r465
http://bato.to/comic/_/comics/zettai-shoujo-seiiki-amnesian-r10454
http://bato.to/comic/_/comics/zero-in-r1781
http://bato.to/comic/_/comics/zero-no-tsukaima-r822
http://bato.to/comic/_/comics/zero-no-tsukaima-zeros-summer-holiday-doujinshi-r5269
The error is the same, but on a different series:
Fetching Zetman
Finished Zetman (00:00:07)
Fetching Zelphy of the Aion
Finished Zelphy of the Aion (00:00:05)
Error: Invalid HTTP response: http://bato.to/comic/_/comics/zero-no-tsukaima-chevalier-r1421
at HttpServiceError (..\mangarack-component-core\src\services\HttpServiceError.ts:13:5)
at Request._callback (..\mangarack-node-core\src\services\httpService.ts:123:16)
at self.callback (../mangarack-node-core/node_modules/request/request.js:198:1)
at emitOne (events.js:77:13)
at Request.emit (events.js:169:7)
at [object Object]._onTimeout (../mangarack-node-core/node_modules/request/request.js:811:1)
at Timer.listOnTimeout (timers.js:89:15)

For some reason I seem to get a similar error on the second link from my list it tries to fetch after a first try (but never the first). I'm using Ubuntu 14.04 server (though it shouldn't have any effect).
I'm using the following script to run it :
#!/bin/bash
mangarack-cli --component.core.batoto.username MyUsername --component.core.batoto.password MyPassword < /home/mereck/list_mangarack.txt
Depending on the links in my .txt file, I get different errors, even within the same list.
At first my list was :
http://bato.to/comic/_/comics/tic-neesan-r3578
http://bato.to/comic/_/comics/prison-school-r1011
http://bato.to/comic/_/comics/sen-to-man-r10545
http://bato.to/comic/_/comics/ore-no-pantsu-ga-nerawareteita-r9786
http://bato.to/comic/_/comics/hatsukoi-lovers-r7599
http://bato.to/comic/_/comics/onii-chan-complex-r7964
http://bato.to/comic/_/comics/oniichan-control-r693
http://bato.to/comic/_/comics/watashi-no-oniichan-r10517
http://bato.to/comic/_/comics/bokurano-r2777
http://bato.to/comic/_/comics/tsurezure-children-r15941
http://bato.to/comic/_/comics/dungeon-meshi-r13871
http://bato.to/comic/_/comics/mahou-tsukai-no-yome-r10801
http://bato.to/comic/_/comics/overlord-r14048
http://bato.to/comic/_/comics/oniichan-no-koto-nanka-zenzen-suki-ja-nai-n-da-kara-ne-r673
http://bato.to/comic/_/comics/gakkou-gurashi-r9554
http://mangafox.me/manga/monster_musume_no_iru_nichijou/
http://bato.to/comic/_/comics/seitokai-yakuindomo-r1597
http://bato.to/comic/_/comics/yandere-kanojo-r489
It stopped working on the 17th chapter of Prison School (didn't log it unfortunately as I tried to figure out how to make it work)
Then, rerunning the script made it stop just after fetching / finishing the first one (+ Tic Neesan).
I tried to put Prison School near the end, but it still wouldn't go past +Tic Neesan.
Then I tried something else and put +Tic Neesan at the end as well, and it worked once again. The list now looks like this :
http://bato.to/comic/_/comics/sen-to-man-r10545
http://bato.to/comic/_/comics/ore-no-pantsu-ga-nerawareteita-r9786
http://bato.to/comic/_/comics/hatsukoi-lovers-r7599
http://bato.to/comic/_/comics/onii-chan-complex-r7964
http://bato.to/comic/_/comics/oniichan-control-r693
http://bato.to/comic/_/comics/watashi-no-oniichan-r10517
http://bato.to/comic/_/comics/bokurano-r2777
http://bato.to/comic/_/comics/tsurezure-children-r15941
http://bato.to/comic/_/comics/dungeon-meshi-r13871
http://bato.to/comic/_/comics/mahou-tsukai-no-yome-r10801
http://bato.to/comic/_/comics/overlord-r14048
http://bato.to/comic/_/comics/oniichan-no-koto-nanka-zenzen-suki-ja-nai-n-da-kara-ne-r673
http://bato.to/comic/_/comics/gakkou-gurashi-r9554
http://bato.to/comic/_/comics/tic-neesan-r3578
http://bato.to/comic/_/comics/prison-school-r1011
http://mangafox.me/manga/monster_musume_no_iru_nichijou/
http://bato.to/comic/_/comics/seitokai-yakuindomo-r1597
http://bato.to/comic/_/comics/yandere-kanojo-r489
It worked through 7 series and stopped at the 37th chapter of Bokurano, then I got stuck with the same error. Trying to rerun the script made it stop at the first series once again :
Fetching Bokurano V07 #036.cbz
Finished Bokurano V07 #036.cbz (00:00:57)
Fetching Bokurano V07 #037.cbz
Error: Invalid HTTP response: http://bato.to/areader?id=6285f42fea726a67&p=10
at HttpServiceError (..\mangarack-component-core\src\services\HttpServiceError.ts:12:81)
at Request._callback (..\mangarack-node-core\src\services\httpService.ts:123:16)
at Request.self.callback (../mangarack-node-core/node_modules/request/request.js:198:1)
at emitTwo (events.js:87:13)
at Request.emit (events.js:172:7)
at Request. (../mangarack-node-core/node_modules/request/request.js:1082:1)
at emitOne (events.js:82:20)
at Request.emit (events.js:169:7)
at Gunzip. (../mangarack-node-core/node_modules/request/request.js:1009:1)
at emitNone (events.js:72:20)
at Gunzip.emit (events.js:166:7)
mereck@Akigumo:~/mangarack$ ./mangarack.sh
mangarack-cli (4.0.1)
Fetching Sen to Man
Finished Sen to Man (00:00:06)
Error: Invalid HTTP response: http://bato.to/comic/_/comics/ore-no-pantsu-ga-nerawareteita-r9786
at HttpServiceError (..\mangarack-component-core\src\services\HttpServiceError.ts:12:81)
at Request._callback (..\mangarack-node-core\src\services\httpService.ts:123:16)
at self.callback (../mangarack-node-core/node_modules/request/request.js:198:1)
at emitOne (events.js:77:13)
at Request.emit (events.js:169:7)
at [object Object]._onTimeout (../mangarack-node-core/node_modules/request/request.js:811:1)
at Timer.listOnTimeout (timers.js:92:15)

I've added some debugging code locally and am in the processing of running several lists through the modified mangarack-cli. Since this is really hard to reproduce, I hope this method gives me enough information to work with. I, too, have seen this error occasionally, and haven't been able to track it down yet. If this doesn't work quite as intended, I'll work on implementing a --debug flag so you can make dumps as well.

I'm beginning to suspect it might have something to do with my internet connection with bato.to (either on my side, in between or on theirs)...
Gave it another try since it was stuck on Prison School. Stopped working on the 170th chapter with this error :
Error: Invalid HTTP response: http://bato.to/areader?id=5304bfc952181f54&p=2
at HttpServiceError (..\mangarack-component-core\src\services\HttpServiceError.ts:12:81)
at Request._callback (..\mangarack-node-core\src\services\httpService.ts:123:16)
at Request.self.callback (../mangarack-node-core/node_modules/request/request.js:198:1)
at emitTwo (events.js:87:13)
at Request.emit (events.js:172:7)
at Request. (../mangarack-node-core/node_modules/request/request.js:1082:1)
at emitOne (events.js:82:20)
at Request.emit (events.js:169:7)
at Gunzip. (../mangarack-node-core/node_modules/request/request.js:1009:1)
at emitNone (events.js:72:20)
Tried again, got immediately stuck with error :
Error: Invalid HTTP response: http://bato.to/comic/_/comics/prison-school-r1011
at HttpServiceError (..\mangarack-component-core\src\services\HttpServiceError.ts:12:81)
at Request._callback (..\mangarack-node-core\src\services\httpService.ts:123:16)
at self.callback (../mangarack-node-core/node_modules/request/request.js:198:1)
at emitOne (events.js:77:13)
at Request.emit (events.js:169:7)
at [object Object]._onTimeout (../mangarack-node-core/node_modules/request/request.js:811:1)
at Timer.listOnTimeout (timers.js:92:15)
Tried 2 more times, first one gave the same error as the last, second one worked again.

Interesting! I'm suspecting something along those lines too, perhaps even something to do with region-based proxies. I'm in the Netherlands region, so if they are distributing their content across servers, I might be hitting a different one, getting different results. What region are you in (I know Xellos is hitting the server from Italy)?
I'm interested to know what content it spews out when you get the invalid HTTP response. Is it possible to run a HTTP debugger on your end? On Windows you can use Fiddler; unfortunately, I'm not that aware of the Linux-based ones. It would be interesting to see what happens.
And have you experienced this only with batoto, or other sites as well? Might be interesting to run a few series against KissManga and/or MangaFox and see if they exhibit the same issue. Just tack on some random series and see what happens.
I'm leaving mine on overnight to see if I get anything in the morning. ~4h without errors atm..

I'm in France so unfortunately it might be a global European CDN...
I'll try to find one and report back if I can get anything out of it. I'll also try some stuff from Mangafox.

Okay, did some packet capture and here are the results (the packetcapture_tic_neesan.cap is a working one, the packetcapture_prison_school.cap got the usual error, everything is openable in Wireshark, just filter with http if you don't want the other protocols informations).
packet_capture.zip
I now highly suspect it has something to do with batoto's network link as I found out that using a real browser I frequently encountered some really slow loading pages... and seeing how the 10 http get requests from mangarack are going unanswered... well there's quite a high probability this is the culprit.
As a workaround, maybe you could add an option to increase the number of tries before giving up? I don't know how many times Chrome retries before displaying an error page, but since it hasn't appeared yet I assume it's quite higher than 10.
Also, no error so far on Mangafox.

Thanks for the capture. Please delete it now and change your password. It includes sensitive data (e.g. session hijacking is possible and I know your password hash now, meaning I could launch a brute force attempt to uncover the associated password).
In the first capture (prison school) I see that the batoto UK CDN is used and basically doesn't reply. Since the timeout is 10 seconds, with a maximum of 10 retries, that could spell disaster. Still, I don't see all of them in the capture. Might need to tweak a little with the timeout and retry settings.
The second capture looks good. Everything goes through as intended. Did that go wrong?

Well I already changed my password since I knew it would be retrievable (though it's just a dummy account with a one-time password anyway...)
The second capture was a good one, just to compare both results (both came from the same CDN though so I quite don't get why it's not answering properly).
Right now, I don't have any problem with Prison School anymore, but it's Mahou Tsukai no Yome which is getting a 100% error rate... will retry tomorrow.

Sorry for the long absence, until yesterday I was on vacation in UK.
If there is some way I can be useful, like trying a particular download list to see if there is some sort of
consistency, please let me know.

Sorry for the long absence, until yesterday I was on vacation in UK.
If there is some way I can be useful, like trying a particular download list to see if there is some sort of
consistency, please let me know.

Bumping everyone involved. Is this still an issue?
I'm guessing the 5-second timeout is a little too short if bato.to is being hammered.

From e-mail:

Fetching Monster V02 #013.cbz
Error: Invalid HTTP response: http://bato.to/areader?id=6e9b7a4638701c98&p=5
at HttpServiceError (..\mangarack-component-core\src\services\HttpServiceError.ts:12:81)
at Request._callback (..\mangarack-node-core\src\services\httpService.ts:123:16)
at Request.self.callback (../mangarack-node-core/node_modules/request/request.js:198:1)
at emitTwo (events.js:100:13)
at Request.emit (events.js:185:7)
at Request. (../mangarack-node-core/node_modules/request/request.js:1082:1)
at emitOne (events.js:95:20)
at Request.emit (events.js:182:7)
at Gunzip. (../mangarack-node-core/node_modules/request/request.js:1009:1)
at emitNone (events.js:85:20)

Same issue. No probable cause yet other than the aforementioned timeout issue.

Well, how about letting the user chose with an option?

Yes, I just tried (I didn't update my collection for a while) and the response was the same:
Error: Invalid HTTP response: https://bato.to/forums/index.php?app=core&module=global&section=login&do=process
at HttpServiceError (..\mangarack-component-core\src\services\HttpServiceError.ts:13:5)
at Request._callback (..\mangarack-node-core\src\services\httpService.ts:123:16)
at Request.self.callback (../mangarack-node-core/node_modules/request/request.js:198:1)
at emitTwo (events.js:87:13)
at Request.emit (events.js:172:7)
at Request. (../mangarack-node-core/node_modules/request/request.js:1082:1)
at emitOne (events.js:82:20)
at Request.emit (events.js:169:7)
at Gunzip. (../mangarack-node-core/node_modules/request/request.js:1009:1)
at emitNone (events.js:72:20)

I'll be increases the allowed time for a connection to begin. I'll change this (~59 seconds maximum)...
let delayInMilliseconds = 1000;
let maximumAttempts = 10;
let timeoutInMilliseconds = 5000;

To this (~2.6 minutes maximum)..
let delayInMilliseconds = 2000;
let maximumAttempts = 5;
let timeoutInMilliseconds = 30000;

I'll reduce the number of retries in favor of a much more lenient timeout. As for:

Well, how about letting the user chose with an option?

I don't like introducing additional options for the sake of fixing a default value that should accommodate the needs of all users. It only increases system complexity for no real purpose. I'll instead be shifting more towards a more tolerant default value as we learn more of the behavioral characteristics of the various networks and providers.

So.. after some initial publishing trouble, 4.0.7 is out with these enhancements in.
Please let me know if this solves your problems :+1:

Just running a full download of some series, for now it seems to be solved!
I'll confirm and close the issue tonight.. meanwhile @Mereck13579 can try it too!

Since I updated my version of Mangarack (3 days ago) I haven't experienced any new error.

@XellossMetallium Actually, one note to make here, is that Batoto has a maximum-login-attempts per x-time. I don't know what the maximum login attempts is, nor what the time limitation is, but it is there. Try logging in on the site, removing your session, and logging in again. It will fail. This same scenario can happen if you run mangarack-cli a few times directly after one another (and will produce that somewhat ambiguous error). Might this have been the issue you've described?

Yes it can be! In these days I'm updating the entire library (by letter, as it was before) before unifying it in a single list as intended.. So I have actually run some instances one after the other.
Now that I'm taking things slowly it worked flawlessly so far. I'm closing the issue, for good I hope :)


Help to locate the performance bottleneck of Pachyderm

Refer to https://github.com/pachyderm/pachyderm/issues/699 . It seems that I didn't give a clear expression about performance problem, as well as the speculation about FUSE is not correct. So I do some comparison test today and repost this issue.
I run the word count sample in which data is come from the wiki page just like the official sample mentioned.
"cmd": [ "wget",
        "-e", "robots=off",
        "--recursive",
        "--level", "1",
        "--adjust-extension",
        "--no-check-certificate",
        "--no-directories",
        "--directory-prefix",
        "/pfs/out",
        "https://en.wikipedia.org/wiki/Main_Page"

First Equipment: I use AWS m4.Xlarge vm + ubuntu14.04 + Kubernetes 1.2.2 + Pachyderm 1.1.0 to run the wourd count example. Note that there is only one node. The whole process uses 29 minutes and the reduce process uses 24 minutes.
ID                                 OUTPUT                                              STARTED             DURATION             STATE
b80331f28ac956ba308011b7ef969054   wordcount_reduce/605aa77a63f4456f867f2c476aced43d   59 minutes ago      24 minutes           success
105a07fead7878072e178a9440e5134f   wordcount_map/366ed6c782024fe0a58e5b43346ea70f      About an hour ago   5 minutes            success
cdeca10ac3f24292824a5c878542517b   wordcount_input/30e1f6d2f41a42eca350f76f15722d8b    About an hour ago   18 seconds           success

Second Equipment: I use AWS m4.XLarge + ubuntu14.04 to run the example directly and locally, of which the CPU number and memory size is the same with that of Pachyderm node. The  map.go  program and reduce shell script used in Pachyderm sample are also involved in the Local server processing. Finally, the local server uses about 2.5 minutes to finish the task. The reduce task takes the major part. See my time statistical result here.
begin time: Fri Aug  5 07:04:26 2016
cat: ./map: Is a directory
end time: Fri Aug  5 07:06:49 2016

Third Equipment: I use AWS m4.XLarge + ubuntu14.04 + Hadoop2.7.2 to run the example. A typical wordcount java program in Hadoop is used to be compared with Pachyderm Pipeline. Unfortunately, Hadoop uses less than 10 seconds although I set the Hadoop as the single-node-non-distribution mode, i.e only one Java process is used.
In conclusion, since the Hadoop Word Count use a different method to realize counting, also , Hadoop is a mature data platform. Please just forget it. Let's focus on the performance difference between local server and Pachyderm Pipeline (2.5minutes vs 30 minutes) to find the performance bottleneck.
Could you give me some idea about the performance problem? I have excluded some impossible reasons.

Docker Pull : The docker image is build before experiment. And the docker load is so quick.
Sources shortage: Don't be worried about that. The 300 GB disk only spends less than 3%. All of the tmpfs has a large space. The memory size is also enough for that.
Pipeline Programming : All of the execute command follow the example guide strictly.
Service Down : I set a daemon to check whether there is any kubernetes or pachyderm service down during the process. And all services work normally.

Several speculations are given:
1 Filesystem Problem: For a simple computational task, Pachyderm doesn't meet a fatal performance problem.  But in word count example, there are many documents (exactly one word one file), does the frequent file write weaken the performance? Or the docker operation on file is slower than local host?
2  Message Send:  The result of wordcount_map is needed to be sent to wordcount_reduce. As I observe, this process is not done by memory, I am not sure it is fulfilled by DISK file or RESTful API.  Probably this step would cause a time loss.

Note: In the second experiment, i.e the local server experiment. I do not use the Pachyderm nor Hadoop. I launch a new server on AWS, download the original data, execute the map.go program to do map task and execute the find /pfs/wordcount_map -name '*' | while read count; do cat $count | awk '{ sum+=$1} END {print sum}' >/tmp/count; mv /tmp/count /pfs/out/basename $count; done to do reduce task. All these steps are manually fulfilled.  And a timer is set to calculate the processing time.

Based on your first and second experiment, the Pachyderm's job shim seems to be the bottleneck.

Hmm, I'm thinking the most likely culprit here is filesystem latency. If I understand your tests correctly the local version is using a local filesystem while the version in amazon is using s3. That means that the reduce stage needs to do a lot of writes in s3, 1 per word that it counts which would be pretty slow. In the future this will be optimized so that those writes can be packed into a single s3 write which should greatly increase the speed.

@ShengjieLuo thanks for bringing the performance issues to our attention!  So far we've been spending most our engineering efforts on the correctness of the system , and not as much on performance, so I'm sure there's a lot of room for improvement there.
I agree with @jdoliner that the performance difference between experiment 1 and 2 is mostly due to the frequent writes to S3.  In general, Pachyderm is much better at handling large files than handling many small files, as is the case for the wordcount example, where each unique word results in a small file.
As for the performance difference between 2 and 3 (local vs Hadoop), I'd attribute it to the fact that both the map.go program and the reduce bash script make use of the local file system, while the Hadoop program probably does the bulk of the work in memory.  Note that in theory, Pachyderm can run in memory as well: https://github.com/pachyderm/pachyderm/issues/546

@jdoliner Thanks you for your advice. I use general SSD (gp2) disk for AWS VM, also the dick operation is regarded as the most possible bottleneck in our team. I still have some further problems. As you have noticed, Pachyderm will be optimized so that writing operation can be packed into a single S3, does it mean that currently Pachyderm use more than one S3 volume to work even I only setup one docker container to fulfill the reduce task?

A table is given to show the performance test more clearly

Nice table!  But just to be clear, in case someone thinks that Hadoop is 174 times faster than Pachyderm: it's an unfair comparison because the Hadoop setup runs purely in memory, while the Pachyderm setup sends data over the network.  It'd be more fair if the Hadoop setup was also writing data to a remote location.

@ShengjieLuo oh and for your follow-up question: Pachyderm only uses one S3 bucket by default.  It's just that currently each time we write a small file, we are sending a write request over the network.  What @jdoliner was suggesting was that we can pack the write requests into one big write request, so as to reduce latency introduced by the network.

@derekchiang For a single-node Hadoop, the operation is based on memory and local file system. However, let's focus on the difference of local server and Pachyderm first. It's important to give a detailed clarification here.
The aws has an automatic method to deploy Kubernetes cluster, and Pachyderm can be deployed on this five-node kubernetes cluster. In this case, S3 bucket is used to transform the data from one node to the other node.
However in my experiment, I launched an instance on AWS. and launch the kubernetes single-node cluster on this instance. The AWS server plays a role the same with a local server.
 Does S3 problem still exist in this step?

@ShengjieLuo ah, I think I misunderstood what you were saying before.  Somehow I thought that the first experiment involved writing to S3, but from what you just said, that doesn't seem to be the case.
So just to be clear:

In the first experiment, you were running Pachyderm on a single node, with no S3 involved.  This means that Pachyderm was writing data to the local disk.
In the second experiment, you were running the map and reduce programs manually, on a single node.

If that's the case, then what you found out is that adding the Pachyderm layer alone causes the processing time to jump from 2.5 min to 29 min, is that correct?  If that's the case, that's certainly very concerning, and we should be looking into this as soon as possible.

@derekchiang I would repeat the test as soon as possible to verify the test result. Thanks for your help

@ShengjieLuo thanks for taking the time to evaluate Pachyderm!

@derekchiang The experiment has been repeated today, no significant difference between the experiment yesterday.


Improve error handling

Error handling in this library should be improved. Rather than printing errors in the FFI extern function, they should be returned to the caller, and likely some of the error checking logic should also be moved to the caller. See below notes:
          > would it make sense to implement this checks on consumer side? i thought that this is ok to panic, as consumer of this lib should do this validation or use valid key/path

Good question. The issue with panic here is that AFAICT it's not possible to trap or handle a panic across the FFI boundary - the process just abruptly dies with basically no information about what's wrong. Yes, we could try to recreate the error-checking logic on the consumer side, but that's a duplication of code and logic and if the upstream library changes, the consumer logic will not. I'd be okay with removing the "path too short", "path too long", etc. error checking to the consumer but I'd prefer to leave the other errors (failed to convert string, failed to parse path) in place.
In general the error handling here could be improved a bunch - rather than printing errors, they should be returned, but we'd need a mechanism for returning errors across the FFI boundary. I suppose just returning a string explanation along with the null ptr would be sufficient. I'll open an issue so we can follow up on this later.
Originally posted by @lrettig in https://github.com/spacemeshos/ed25519_bip32/issues/5#issuecomment-1523613579

The .map_err(...) could be improved with use of wonderful crates like eyre or anyhow that allow providing context easily and record error backtrace for easier debugging.
Originally posted by @poszu in https://github.com/spacemeshos/spacemesh-sdk/pull/4#discussion_r1205150940


Validation Error

The validation for the NummericUpDown, DatePicker and Autocomplete doesn't work like it should.

@tiimm50 can you please create a BlazorFiddle example of these issues if they still exist. We are trying to close out old issues. Thanks

Merged fix into develop branch.


i have been looking so long for this but im unable to getr it working

I have been wanting to be able to search my notes with rofi for so long. I knew it was possible however I dont know how to write that code.
It looks like this plugin shows tabs and windows but I thought you were able to search notes.
anyway I tried using the script you added. After reading the  other issue where you guys talk about getting it working by putting it on PATH and naming.  but when I did that I keep getting errors ,

script not found,
command not found,
I put the script in bin
and made sure its on path with
script set to executable
permissions are pretty much wide open..
and still nothing . anyway keep up the great work as im sure once i get it going ill enjoy it.

Hey there. I should probably make some step by step instructions to help these issues.
First thing is in the python script, you should modify this part to match your vault:
# Set these two variables with your personal settings
vault_name = 'my-vault'
vault_path = '/home/my-name/my-files/my-vault'

Then a good test from here is to run the python script on it's own like python obsidian-rofi.py and see that it prints stuff.

I added some more instructions in the readme.
Also if you are on nixos, the shebang of the script might need to be #!/usr/bin/python


Inserting string values like "with space" result in "with\ space"

I'm not able to implement the following insert:
INSERT cpu,host=serverA,region=us_west value="with space"
or via REST:
curl -i -XPOST 'http://localhost:8086/write?db=mydb' --data-binary 'cpu,host=server02 value="with space"
Could you verify and post a piece of code to not escape spaces with "\ "?

Looks like an issue with influx and not this library?

The problem is in the class PointFormatter, in this piece of code:
        protected virtual string EscapeNonTagValue(string value)
        {
            Validate.IsNotNull(value, "value");
            var result = value
                // literal backslash escaping is broken
                // https://github.com/influxdb/influxdb/issues/3070
                .Replace(@"\", @"\\")
                .Replace(@"""", @"\""") // TODO: check if this is right or if "" should become \"\"
                .Replace(@" ", @"\ ")
                .Replace(@"=", @"\=")
                .Replace(@",", @"\,");

            return result;
        }

There is an explicit introduction of a backslache before every space.
It's not an InfluxDB issue. Inserting strings containing spaces from command line is working,
and https://github.com/influxdata/influxdb-csharp library for example is not bugged like this.

The thing is, InfluxData.Net has been forked from InfluxDB.Net and that piece of code was inherited from the original library. And I'm not fully sure what specific escape is there.
I tried to comment out that specific line and all the tests I have still passed, but I'm a bit afraid to remove that replace and to deploy this change to production.

Actually, it seems that the documentation for older versions did require escaping. But that then makes it a bit weird that all the tests passed..

More info:

If a tag key, tag value, or field key contains a space , comma ,, or an equals sign = it must be escaped using the backslash character . Backslash characters do not need to be escaped. Commas , and spaces will also need to be escaped for measurements, though equals signs = do not.

Could you please provide me with a piece of C# code which you're using to insert this value so I could fully reproduce your issue and try to do something about it?
Also, for the reference, this is the escaping documentation for 1.2 (I might need it to fix this):
https://docs.influxdata.com/influxdb/v1.2/write_protocols/line_protocol_reference/#special-characters
https://docs.influxdata.com/influxdb/v1.2/write_protocols/line_protocol_reference/#special-characters
I assume you're using InfluxDB v1.2?

I have read it well, I copy and past it here so you can read it better:
Special Characters For tag keys, tag values, and field keys always use a backslash character \ to escape:

commas ,
equal signs =
spaces
For measurements always use a backslash character \ to escape:
commas ,
spaces
For string field values use a backslash character \ to escape:
double quotes "
Line Protocol does not require users to escape the backslash character . Users do not need to escape all other special characters. Examples
Example 1: Write a point with special characters
`

INSERT "measurement\ with\ quo⚡es\ and\ emoji",tag\ key\ with\ sp🚀ces=tag,value,with"commas" field_k\ey="string field value, only " need be esc🍭ped"
`

Hi, thanks!
I assume you used BasicClientModule.QueryAsync() with that insert that you just posted? Or did you use a different lib module?
Thnx

I have the same issue, writing (WriteAsync) textual event data with this lib, result in improper escaped text.
This is a screen shot of a grafana dashboard:

Updated the formatter to match the docs. Not deployed to nuget yet because I have a few more things to update in the library before that. If you need this asap, feel free to use the dev branch.

I'll deploy this shortly as part of nuget v8. If you find any bugs with it, please reopen the case. And thanks everyone for help!


Fix android build

This PR contains following fixes:

removed soversion from libroc.so library
LogLevel class caching during JNI_OnLoad
fix thread attaching and detaching on logger_handler
added internet permissions to AndroidManifest.xml

/resolve #15
/resolve #16

I've tested changes. Fixes for logger and build are working 👍

LGTM


fix: dashboard tooltip alignment

Fixes #3079
Describe the changes you have made in this PR -

added a new class for the center alignment of the tooltip.

Screenshots of the changes (If any) -

Note: Please check Allow edits from maintainers. if you would like us to assist in the PR.

@tachyons @pavanjoshi914 Kindly Review.

Coverage remained the same at 81.785% when pulling 57104f83d1d4e09f177a4f3c4514291e58f621f2 on vedant-jain03:issue3079 into 673f5edf97f64d3593475e1b00db011b165e75d7 on CircuitVerse:master.

can you post screenshot of how alignment of tooltip looks on homepage after this change is done

can you post screenshot of how alignment of tooltip looks on homepage after this change is done

can you post screenshot of how alignment of tooltip looks on homepage after this change is done

@pavanjoshi914 I have made a new class for alignment of the tooltip as a center and I included that class only in dashboard Project div. In this way we can separate tooltip alignment for both of our Project Div without modifying tooltip styling.

LGTM


Make accessible as trait

Thanks for making SubStringy!!!
For my project, I will be converting SubStringy to a trait and using it in my own Stringy class.
I suggest you do the same, and keep a SubStringy class around that implements that trait to have the best of both worlds.
I could just use SubStringy but that would conflict with other Stringy extensions.
This would be the rudiment of a Stringy extension econsystem, and you'd be first ;)

Hi @carlocapocasa thanks for the feedback. I actually was thinking about this some time ago, but never implemented. Can you do a PR with the code converted to a trait? I would be glad to accept it! Don't forget to fix this unit tests! ;)

@carlocapocasa feature available since d71aafa.


is with loader

When converting a ti sdk app to use this module I keep getting this error when following the instructions on the site: Couldn't find module: node_modules/ti-commonjs/lib/ti-commonjs for architecture: x86_64.  I don't know where I am supposed to put the node_modules folder, it exists in /Resources, /app/lib, and /.  What am I missing?  Any help on resolving this issue would be appreciated as I am trying to use this so I can use wiredep for dependency injection.
Thanks in advance.

It should work correctly if you install it with
npm install ti-commonjs --prefix ./app/lib

that means that you should find it under ./app/lib/node_modules/ti-commonjs.
Remember that this module is only for Alloy projects.
If you need something similar without Alloy then use titaniumifier as a plugin.


Binance US is still using old rate limits

Describe the bug
I'm running a few bots on same IP and recently got banned because violating binance.us rate limits. From the log I think binance.us is using old rate limits while hummingbot is using new one. My work around is just disabling binance.us from rate oracle since I don't need that info.
I'm not running the latest version but I believe this issue also affect latest one.  Raising this so people can be aware of the issue.
Steps to reproduce

Run a few bots on binance exchange
Observe the log

Release version
1.21.0
Type of installation
Docker
Attach required files
There was an error requesting exchange info.
Traceback (most recent call last):
  File "/home/hummingbot/hummingbot/connector/exchange_py_base.py", line 1019, in _initialize_trading_pair_symbol_map
    exchange_info = await self._api_get(path_url=self.trading_pairs_request_path)
  File "/home/hummingbot/hummingbot/connector/exchange_py_base.py", line 846, in _api_get
    return await self._api_request(*args, **kwargs)
  File "/home/hummingbot/hummingbot/connector/exchange_py_base.py", line 879, in _api_request
    request_result = await rest_assistant.execute_request(
  File "/home/hummingbot/hummingbot/core/web_assistant/rest_assistant.py", line 78, in execute_request
    raise ioe
OSError: Error executing request GET https://api.binance.us/api/v3/exchangeInfo. HTTP status is 429. Error: {"code":-1003,"msg":"Too much request weight used; current limit is 1200 request weight per 1 MINUTE. Please use WebSocket Streams for live updates to avoid polling the API."}

2024-05-07 02:52:51.334 There was an error requesting exchange info.
Traceback (most recent call last):
  File "/home/hummingbot/hummingbot/connector/exchange_py_base.py", line 1019, in _initialize_trading_pair_symbol_map
    exchange_info = await self._api_get(path_url=self.trading_pairs_request_path)
  File "/home/hummingbot/hummingbot/connector/exchange_py_base.py", line 846, in _api_get
    return await self._api_request(*args, **kwargs)
  File "/home/hummingbot/hummingbot/connector/exchange_py_base.py", line 879, in _api_request
    request_result = await rest_assistant.execute_request(
  File "/home/hummingbot/hummingbot/core/web_assistant/rest_assistant.py", line 78, in execute_request
    raise ioe
OSError: Error executing request GET https://api.binance.us/api/v3/exchangeInfo. HTTP status is 418. Error: {"code":-1003,"msg":"Way too much request weight used; IP banned until<PHONE_NUMBER>547. Please use WebSocket Streams for live updates to avoid bans."}

Hey @sytranvn, this issue has been resolved in the latest release. We've split the binance and binance.us into separated oracle.

@mlguys great! Please close this issue.


Split tall images into horizontal chunks

Currently, when a thread is very long, the render becomes hard to read:

Add a feature to split the image horizontally.

I've had some time to think about this, and imo the best course of action would actually be to just roll up the post above a specific height (roughly calculated from the height of the individual elements), and re-use the unroll function for those who want to see the whole post regardless.

Another idea I had related to this - for long posts, we could add a "buffer" page that is just the rendered thread with some minor changes (the page is centered, video embeds actually work, use iframe embeds, etc.) and a link to redirect to the Tumblr post. This way users don't have to log in to see the whole post before the login-wall kicks in. (We could also make this a default option by either setting an URL parameter or through an user preference stored in the browser cookies; FixTweet has something similar for some options, could be cool to add a settings page.)


Multi-targeting and assembly signing

Very cool library. 👍
Gonna use it to enable more covariance in my project TypedRest.
This PR enables:

multi-targeting (.NET Standard 1.0, .NET Standard 2.0 and .NET Framework 4.5) - This reduces the number of transitive dependencies pulled in by NuGet on some platforms.
assembly singing - for those old fashioned .NET Framework projects that still need it. :)

Looks good, thanks!


Anthropic Cluade 2.1 Support

Anthropic just released Cluade 2.1 API Calling, they recommand use claude-2.1 and claude-2.0 to call the model, instead of claude-2.

Also, it support system prompt, so, may be we can process it as asme as OpenAI API.

Done


Multiple unexpected results occur when uploading/downloading large VHD files

Storage Explorer Version: 1.5.0
Platform/OS Version: Windows 10/ Linux Ubuntu 16.04/ MacOS High Sierra
Architecture: ia32/x64
Build Number: 20180927.3
Commit: 4280a93
Regression From: Not a regression
Steps to Reproduce:

Navigate to 'Experimental' menu -> Check 'Use AzCopy for Improved Blob Upload and Download'.
Open one premium blob container and try to upload a large vhd file (Like 30G) to it.
Check the result.

Expected Experience:
The large vhd file can be uploaded successfully.
Actual Experience:
There are three unexpected results for the uploading/downloading operation.

Fail to upload with the error 'AzCopy quit unexpectedly'.

The uploading activity is cancelled automatically.
The uploading activity can't be completed.

Hi @MRayermannMSFT ,We verified this issue on the build 20181022.4. The uploaded large VHD(127G) can be uploaded successfully, but the percentage number stayed at 12% for a long time. And continue to waiting about 2 hours until the the activity is complete.

@v-xuanzh , should we close this issue?

Hi @MRayermannMSFT ,We verified this issue on the latest build, There is one result when uploading a large file to one premium blob container.
Result Experience:
The uploading status has always been 7% and it cannot be completed. And I'm not sure if it is a network issue. ☹

This issue doesn't reproduce on build 20191220.9. So we close it.