Some unexpected problems trying to scrape a screenshot

The first issue is that after using this snippet to create a use-once agent:
const myagent = new Agent({
  userAgent: ua,
  viewport: {
    screenHeight: 1024,
    screenWidth: 768,
    height: 1024,
    width: 768,
  }
});

and then when done scraping calling:
await myagent.close();

does work once after starting my node service that runs this function, but
subsequent times the same function is called, I get this error in node console:
2021-08-04T18:32:42.557Z ERROR [/Users/andy/repos/test-repo/app/node_modules/@secret-agent/client/connections/ConnectionFactory] Error connecting to core {
  error: 'Error: connect ECONNREFUSED <IP_ADDRESS>:63738',
  context: {},
  sessionId: null,
  sessionName: undefined
} Error: connect ECONNREFUSED <IP_ADDRESS>:63738
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1146:16) {
  errno: -61,
  code: 'ECONNREFUSED',
  syscall: 'connect',
  address: '<IP_ADDRESS>',
  port: 63738
}

The second surprise is the screenshot itself, taken with:
const scrollHeight: number = await myagent.executeJs(() => {
  return document.scrollingElement.scrollHeight;
});

let buffer: Buffer;
buffer = await myagent.takeScreenshot({
  format: 'png',
  rectangle: {
    scale: 1,
    height: scrollHeight,
    width: 1024,
    x: 0,
    y: 0,
  }
});

I used this url: https://www.whatsmyua.info
The visible text in the screenshot is not centered as one would expect for the page I used, but is more or less
left-justified, and a large portion of the page is clipped even though I used the scrollHeight, which I checked
had not grown after taking the screenshot.

The third problem is that if I call takeScreenshot this way it fails with an error, even though typescript tells me rectangle is optional:
buffer = await myagent.takeScreenshot({
  format: 'png',
});

Hope I didn't do something stupid!

On 1, I found that this happens for some reason when creating an http server that calls my test scraping function, and not when I call it more than once consecutive times in the same nodejs "thread".
so here is my test function's typescript file:
/* eslint-disable no-console */
import { Agent } from 'secret-agent';
import ExecuteJsPlugin from '@secret-agent/execute-js-plugin';
import * as fs from 'fs';

const ua = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_16_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.165 Safari/537.36';

export async function testUrl(requestUrl: string, imageName: string): Promise<void> {
  const myagent = new Agent({
    userAgent: ua,
    viewport: {
      screenHeight: 1024,
      screenWidth: 768,
      height: 1024,
      width: 768,
    }
  });

  try {
    myagent.use(ExecuteJsPlugin);
    await myagent.goto(requestUrl);
    await myagent.waitForPaintingStable();

    const getScrollHeight = async (): Promise<number> => {
      // eslint-disable-next-line @typescript-eslint/ban-ts-comment
      // @ts-ignore
      const scrollHeight: number = await myagent.executeJs(() => {
        // eslint-disable-next-line @typescript-eslint/ban-ts-comment
        // @ts-ignore
        return document.scrollingElement.scrollHeight;
      });
      console.log('scrollHeight', scrollHeight, 'for', requestUrl);
      return scrollHeight;
    };

    const takeScreenshot = async (scrollHeight: number): Promise<void> => {
      const buffer: Buffer = await myagent.takeScreenshot({
        format: 'png',
        rectangle: {
          scale: 1,
          height: Math.max(scrollHeight, 768),
          width: 1024,
          x: 0,
          y: 0,
        }
      });
      fs.writeFileSync('../screenshots/' + imageName + '.png', buffer, 'binary');
    };

    const height1 = await getScrollHeight();
    await takeScreenshot(height1);

    const height2 = await getScrollHeight();
    if (height2 > height1) {
      console.log('oops: scrollHeight increased after taking screenshot.  Taken too soon?');
      await takeScreenshot(height2);
    }

    console.log(requestUrl + ' ok');
  } finally {
    try {
      await myagent.close();
    } catch (e) {
      console.log('unexpected error closing new Agent', e);
    }
  }
}

// (async() => {
//   await testUrl('https://example.org', 'example');
//   await testUrl('https://www.whatsmyua.info', 'myua');
// })();

and here is my http server written in javascript that compiles the typescript function above:
/* eslint-disable prefer-template */
/* eslint-disable no-console */
const _http = require('http');
const { testUrl } = require('./test');

function ProcessExit(num) {
  process.exit(num);
}

function TextResponse(res, txt) {
  res.writeHead(200, {
    'Content-Type': 'text/plain'
  });
  res.write(txt);
  res.end();
}

function StartServer() {
  console.log('listening for scrape requests');

  const server = _http.createServer((req, res) => {
    let data = '';
    req.on('data', (chunk) => {
      data += chunk;
    });
    req.on('end', () => {
      let json;
      let err;
      try {
        json = JSON.parse(data);
      } catch (e) {
        err = e;
      }
      if (err) {
        console.log('could not parse json request:', err);
        TextResponse(res, 'could not parse json request: ' + err);
      } else if (!json) {
        TextResponse(res, 'falsy json request: ' + json);
      } else if (typeof json.requestUrl !== 'string') {
        TextResponse(res, 'json.requestUrl invalid not a string');
      } else if (typeof json.imageName !== 'string') {
        TextResponse(res, 'json.imageName invalid not a string');
      } else {
        (async() => {
          let err2;
          try {
            await testUrl(json.requestUrl, json.imageName);
          } catch (e) {
            err2 = e;
          }
          if (err2) {
            console.log('testUrl failed:', err2);
            TextResponse(res, 'testUrl failed: ' + err2);
          } else {
            TextResponse(res, 'created image in server: ' + json.imageName);
          }
        })();
      }
    });
  });

  server.setTimeout(0);
  server.listen(8888);
}

(async() => {
  try {
    StartServer();
  } catch (e) {
    console.log(e);
    if (e.stack)
      console.log('' + e.stack);
    ProcessExit(1);
  }
})();

I'll have to figure out how to take a session trace in a little while if you still need that.

NOTE: in my testUrl function above, for this pretty simple webpage, it seems like waitForPaintingStable() didn't work as well as it should because on my machine, the scrollHeight obtained after waitForPaintingStable was 1413, but then when after taking the screenshot and writing it to a file, when I asked for scrollHeight again, it was 1971, prompting me to "retake" the screenshot, to make sure that wasn't part of why the screenshot is clipped.

Here's the session db:
sessions.db.zip

I think what's happening is you are using the default "full-client" SecretAgent "connection" which is built for single use scrapes, but I think you're triggering the auto-shutdown when you call close the first time (think about booting up a script and then wanting the whole thing to tear down when you close). I think you'll get more reliable behavior by spinning up a CoreServer and then pointing your agents at the persistent server (SecretAgent already comes with a client/server setup - https://secretagent.dev/docs/advanced/remote). You can run the server in the same process as your existing server if you want - doesn't have to be a separate process.
Regarding paintingStable - that event is specifically geared around the page being visible above the fold, not "all content loaded". You can add a "domContentLoaded" trigger to wait for the page to be fully "loaded" as well.
With your screenshot, it seems like your viewport width & height are mismatched in your screenshot rectangle. Could that be why it's showing up with a strange shape? I guess it doesn't explain the x/y..

Thanks for the help.  You were right about the viewport having switched the width and height.  However, after fixing my code with everything you mentioned above, the screenshot still is clipped even though the height specified in takeScreenshot is always the full scrollHeight of 1971 for this url.
I don't see anything else that could explain the clipping, and in fact, now it appears that though the scrollHeight is 1971, and indeed the screenshot image height is 1971, and includes the proper background for the full 1971, somehow the text content inside the dom looks like it is being clipped to the viewports height of 768.  Is this possible?
(Here's the fixed code)
/* eslint-disable no-console */
import { Agent, ConnectionFactory, ConnectionToCore, LocationStatus } from 'secret-agent';
import ExecuteJsPlugin from '@secret-agent/execute-js-plugin';
import * as fs from 'fs';

const ua = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_16_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.165 Safari/537.36';

let sharedConnection: ConnectionToCore = null;

function getConnection (): ConnectionToCore {
  if (sharedConnection !== null) return sharedConnection;
  sharedConnection = ConnectionFactory.createConnection({
    maxConcurrency: 4,
  });
  return sharedConnection;
}

export async function testUrl(requestUrl: string, imageName: string): Promise<void> {
  const myagent = new Agent({
    userAgent: ua,
    viewport: {
      screenHeight: 768,
      screenWidth: 1024,
      height: 768,
      width: 1024,
    },
    connectionToCore: getConnection(),
  });

  try {
    myagent.use(ExecuteJsPlugin);
    await myagent.goto(requestUrl);
    await myagent.waitForPaintingStable();
    await myagent.activeTab.waitForLoad(LocationStatus.DomContentLoaded);

    const getScrollHeight = async (): Promise<number> => {
      // @ts-ignore
      const scrollHeight: number = await myagent.executeJs(() => {
        // @ts-ignore
        return document.scrollingElement.scrollHeight;
      });
      console.log('scrollHeight', scrollHeight, 'for', requestUrl);
      return scrollHeight;
    };

    const takeScreenshot = async (scrollHeight: number): Promise<void> => {
      const buffer: Buffer = await myagent.takeScreenshot({
        format: 'png',
        rectangle: {
          scale: 1,
          height: Math.max(scrollHeight, 768),
          width: 1024,
          x: 0,
          y: 0,
        }
      });
      fs.writeFileSync('../screenshots/' + imageName + '.png', buffer, 'binary');
    };

    const height1 = await getScrollHeight();
    await takeScreenshot(height1);
    console.log(requestUrl + ' ok');
  } finally {
    try {
      await myagent.close();
    } catch (e) {
      console.log('unexpected error closing new Agent', e);
    }
  }
}

Can you see if the latest version helps your screenshot issue if you provide no rectangle?

Scratch that. I see it happening. No need to try it

NOTE for implementation.. Looks like in Chromium, you have to change the visualViewport to take a full page screenshot then restore it. We need to think about how we should think about this from a detection perspective.

Hi, I was wondering if this has turned out to be difficult to fix from the standpoint of bot detection, since I noticed that the behavior is still the same as of the latest version.  What exactly would be the detection exposure if a quick-and-dirty fix were to be done?  Is it possible that you could point us to an easy approach and we could take the risk of detection ourselves in some kind of plugin?

@andynuss - I just haven't gotten to this. There's a lot of stuff on the plate to do, and this one just hasn't made it to the top of the priorities yet. You could give a plugin a try or a PR - I think for a plugin, you'd just want to be able to set the page to the full length of the page (here's how puppeteer does that: https://github.com/puppeteer/puppeteer/blob/327282e0475b1b680471cce6b9e74ecc14fd6536/src/common/Page.ts#L2664)


TagWithSpaces does not look at suite settings 'Force Tags' and 'Default Tags'

TagWithSpaces only looks at the [Tags] setting for a test.
It does not look at 'Force Tags' or 'Default Tags'.
It may be worth documenting that on the description of the rule.

Thanks for reporting this. I'll take a look at it.
On Tue, May 9, 2017 at 9:02 AM, Nicholas Daley<EMAIL_ADDRESS>wrote:

TagWithSpaces only looks at the [Tags] setting for a test.
It does not look at 'Force Tags' or 'Default Tags'.
It may be worth documenting that on the description of the rule.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/boakley/robotframework-lint/issues/40, or mute the
thread
https://github.com/notifications/unsubscribe-auth/ABEmYhOl8xZsnysZU3Nmw1FMOpsydfyhks5r4HIAgaJpZM4NVXWl
.


Serialize uses new file path format when using namespaces which causes Linter issues

Just updated to the Latest AG version. The serialization of interfaces was changed as it now generates a new file path. The namespace is now enclosed in round brackets whereas before it was enclosed with hash.

The class paths are still the same.

When the linter now tries to check, it get this error (beside the object naming errors):

If this naming change still in progress or was it not meant to be this way?

Thx, the experimental flag was still on. Removed it and serialization was back to the old format.


ValueError: cannot convert float NaN to integer in loss_string += [f"Avg Loss {species}: {round(np.average(epoch_ave_losses[species]))}"]

Hi Yanay,
I'm running the tutorial /Vignettes/frog_zebrafish_embryogenesis/Train SATURN.ipynb after successfully running the dataloader notebook.
However when I run the command I get this error:
Pretraining...
0%|                                                   | 0/200 [01:01<?, ?it/s]
Traceback (most recent call last):
File "/Users/mdallang/unil/workspace/SATURN/Vignettes/frog_zebrafish_embryogenesis/../../train-saturn.py", line 1072, in
trainer(args)
File "/Users/mdallang/unil/workspace/SATURN/Vignettes/frog_zebrafish_embryogenesis/../../train-saturn.py", line 654, in trainer
pretrain_model = pretrain_saturn(pretrain_model, pretrain_loader, optim_pretrain,
File "/Users/mdallang/unil/workspace/SATURN/Vignettes/frog_zebrafish_embryogenesis/../../train-saturn.py", line 258, in pretrain_saturn
loss_string += [f"Avg Loss {species}: {round(np.average(epoch_ave_losses[species]))}"]
ValueError: cannot convert float NaN to integer
I changed the path to the protein embeddings and the torch.device line (I replace mps for cuda since I have a mac) in the script train-saturn.py. Beside this I didn't do any change.
I tried to debug it by adding a line of code which prints the value of loss for each epoch before calculating the average: print(f"Epoch {epoch} - {species} losses: {epoch_ave_losses[species]}").
 On the other side, Rank Loss is a number and always different. So I was wondering why calculating the average loss for each epoch (epoch_ave_losses) and if I can skip this line of code.
I tried to use np.nan_to_num to replace nan with zero before calculating the average:
clean_losses = np.nan_to_num(epoch_ave_losses[species], nan=0.0)
loss_string += [f"Avg Loss {species}: {round(np.average(clean_losses))}"]
The code then run, but I end up having another error at the STARTING METRIC TRAINING
Traceback (most recent call last):
File "/Users/mdallang/unil/workspace/SATURN/Vignettes/frog_zebrafish_embryogenesis/../../train-saturn.py", line 1079, in
trainer(args)
File "/Users/mdallang/unil/workspace/SATURN/Vignettes/frog_zebrafish_embryogenesis/../../train-saturn.py", line 799, in trainer
train(metric_model, loss_func, mining_func, device,
File "/Users/mdallang/unil/workspace/SATURN/Vignettes/frog_zebrafish_embryogenesis/../../train-saturn.py", line 131, in train
loss.backward()
File "/opt/miniconda3/envs/saturn/lib/python3.10/site-packages/torch/_tensor.py", line 525, in backward
torch.autograd.backward(
File "/opt/miniconda3/envs/saturn/lib/python3.10/site-packages/torch/autograd/init.py", line 267, in backward
_engine_run_backward(
File "/opt/miniconda3/envs/saturn/lib/python3.10/site-packages/torch/autograd/graph.py", line 744, in _engine_run_backward
return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: Function 'NativeLayerNormBackward0' returned nan values in its 0th output.
I imagine all this is due to these nan values?
I hope the problem is clear, I went through all the issues open to see if someone already raised one similar before asking, but I didn't find any.
Thank you,
Michela

Hi Michela,
There shouldn't be any NaN's during training, it's possible that the change to MPS could have caused this but I am not sure.
Any kind of NaN during training is really concerning and will probably completely throw off the model.

Hi Yanay,
Thank you for answering. I run the model in cpu and the epoch loss is fine, there aren't nan values anymore. So you were right, the problem was the change to MPS, even though I don't understand the reason behind it :)
If it's too slow, I will run it in the cluster where we have cuda installed!


Update /ja/docs/reference/command-line-tools-reference/feature-gates/

This is a Feature Request

What would you like to be added

Why is this needed
Comments

/assign

/milestone 1.16-ja.1

/language ja

/close


virtualenvwrapper plugin fail to deactivate virtualenv when parent directory is a git repository

Consider the following directory layout:
/home/src
/home/src/gitrepo => git repository
/home/src/gitrepo/adirwithvirtualenv => a repository with a virtualenv ( either a .venv or else ).
When going from adirwithvirtualenv to gitrepo ( cd .. ), deactivate is not called because $ENV_NAME is filled with the name of the directory, as it is a git repository.

Pull request: #4604

I think this is intended behavior. Git repos, even without venv-specific customizations, are considered to be project roots, so when this happens, it should switch you from the adirwithvirtualenv venv to the gitrepo venv. I think just calling workon without deactivate first will accomplish that. This change looks like it would end up fully deactivating the venvs in the example case. (Is that what it's intended to do? Or should deactivate just be called before activating the new environment?)
Have a look at the discussion in #4122, #4110, and #3918. There was a while when virtualenvwrapper did not consider plain git repos to be project roots, but this was a behavior accidentally introduced by #3918.
Maybe you'd want to add a setting that controls whether ordinary git repos are considered project roots? At any case, we should probably run this by the other virtualenvwrapper users before merging, since they've already had some unexpected behavior changes this year.

In my case, gitrepo is not a virtualenv. Hence the annoying behaviour.
But yes, it may be a change for some people, thanks for the headsup !
Pull request has been updated to add an option DEACTIVATE_VENV_ASAP to control that behaviour.
Without setting that option, the behaviour is the same as previously.

Cool.
Maybe name it VENV_DEACTIVATE_ASAP so it's prefixed with the plugin name? We're starting to get a lot of config variables floating around, and that might help keep it organized.

What's the status of this issue? It would appear that this is a problem not only when the parent directory is a git repo but any time the destination directory of the cd command is. (see referenced issue above)

I just submitted a PR with my stab at this issue. I took a different approach which also resolves another issue (#5816). Any thoughts are much appreciated.


Custom Nodes are not getting a connection line between them when a new node is created

Describe the Bug
I have created Custom Nodes for my company's  usecase. Our goal is to build a workflow builder. However, we felt the example provided was not exactly what we had in mind, so I set up our flow diagram based off the Add Node on Edge Drop example.
So far for the most part, things are working fine, but I have a couple of issues that I wish to raise, with the most important one being the fact that when I create a new node, the handles of the source and target node are not connected. I get the following warning message:
[React Flow]: Couldn't create edge for source handle id: 0-source-handle}; edge id: 0 => 1.
There was a link attached which explains the error, and I did take a look at it, but I'm not sure what is wrong with the logic in my sandbox.
Here's how my flow diagram looks on the sandbox:

Additionally, I have the following question:
How do I reduce the zoom of the flow diagram by default?  The default zoom feels quite zoomed in. I tried using the defaultViewport property like this:
const defaultViewPortOptions = {   x: 0,   y: -300,   zoom: 0.25 };
However, this does not change the zoom. Specifying min and max zoom values to the fitViewOptions object however changes it for me. What is the purpose of the defaultViewport property then? Could you clarify this for me please?
I would also like to know what property I have to specify so that my Start Node by default is positioned higher up the screen. I know i can do this by setting the position value in the initialNodes property for this node, but isn't it possible to do it with defaultViewport?
Your Example Website or App
https://codesandbox.io/s/exciting-sky-xxq3z4
Steps to Reproduce the Bug or Issue

Go to my sandbox
On the UI screen, there is a default node called Start. Click on the plus icon
It creates a new node, but the two nodes do not get connected, but I instead get a warning : [React Flow]: Couldn't create edge for source handle id: 0-source-handle}; edge id: 0 => 1.

Expected behavior
I expected that my Custom Nodes would be connected, but they are not.
Screenshots or Videos
No response
Platform

OS: Linux (Ubuntu 20.04
Browser: Google Chrome
Version: 105.0.5195.125

Additional context
No response

I forgot to mention this but I am using the latest version of Reactflow v11.

Hi @kash0296, please see my codesandbox with edits: https://codesandbox.io/s/nice-violet-fpyimh?file=/src/App.js

The reason why the edge could not be drawn by React Flow was that the source and target handles where not rendered when your custom node was not connected (https://codesandbox.io/s/exciting-sky-xxq3z4?file=/src/CustomNodes/StarterNode.jsx:231-267), so the connection could not be made. I have removed the condition to render the handles so that they are rendered all the time. I am assuming you want to hide the handles as long as the node is not connected. Therefore, I would recommend to do this via CSS, for example: style={{ opacity: data.isConnected ? 1 : 0 }}.
The defaultViewPort did not have any effect because you are setting the fitView prop. The fitView prop tells React Flow to center all elements in the viewport on load. To have more control on the initial view, you can remove the fitView prop and pass your configuration with defaultViewPort (see my sandbox as example). Also notice that you might want to set minZoom and maxZoom as this limits the zoom by default.
For displaying the nodes more up the screen, I would recommend adjusting the defaultViewPort prop instead of moving the nodes with their position individually.

Hope that helps!

@chrtze Thank you for a quick response! I really appreciate the help. This works for me. Should I face any further issues with anything else, I will post on the issues thread if I don't already find a solution from other posts.
Thanks again!


Enable downloading of protected content. See issue #979

Please see issue #979

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

Thanks for the PR Simon! This looks good but perhaps we should have a test to assert that only the desired headers are removed. Could you write a test before we merge this?

I'll see what I can do.

@mholt Sorry for the delay. Tests added.

Nice! Thanks for your contribution, @slightfoot! We'll give it a go.


Running on node v6.6

I got this error
{"actid":"ovlusnyo3je3/ry6yiqc31j49","msg":{"type":"web","host":"<IP_ADDRESS>","path":"/act","protocol":"http","timeout":5555,"max_listen_attempts":11,"attempt_delay":222,"serverOptions":{},"pin":"salestax","model":"actor","msgprefix":"seneca_","callmax":111111,"msgidlen":12,"role":"transport","hook":"listen","plugin$":{"name":"mesh","tag":"-"},"tx$":"ry6yiqc31j49","meta$":{"id":"z1guu98rgcsg/ry6yiqc31j49","tx":"ry6yiqc31j49","pattern":"hook:listen,role:transport,type:web","action":"(aax0e24x8tol)","plugin_name":"transport","plugin_tag":"-","prior":{"chain":[],"entry":true,"depth":0},"start":1482487829534,"sync":true}},"entry":true,"prior":[],"meta":{"plugin_name":"transport","plugin_tag":"-","plugin_fullname":"transport","raw":{"role":"transport","hook":"listen","type":"web"},"sub":false,"client":false,"args":{"role":"transport","hook":"listen","type":"web"},"rules":{},"id":"(aax0e24x8tol)","pattern":"hook:listen,role:transport,type:web","msgcanon":{"hook":"listen","role":"transport","type":"web"},"priorpath":""},"client":false,"listen":false,"transport":{},"kind":"act","case":"ERR","info":"Expected \":\" but \"}\" found.","code":"act_callback","err":{"expected":[{"type":"literal","value":":","description":"\":\""}],"found":"}","offset":9,"line":1,"column":10,"name":"SyntaxError","eraro":true,"orig":{"message":"Expected \":\" but \"}\" found.","expected":[{"type":"literal","value":":","description":"\":\""}],"found":"}","offset":9,"line":1,"column":10,"name":"SyntaxError"},"code":"act_callback","seneca":true,"package":"seneca","msg":"seneca: Action hook:listen,role:transport,type:web callback threw: Expected \":\" but \"}\" found..","details":{"message":"Expected \":\" but \"}\" found.","pattern":"hook:listen,role:transport,type:web","instance":"Seneca/3zsswehvo3im/1482487829211/40722/3.2.2/-","orig$":{"message":"Expected \":\" but \"}\" found.","expected":[{"type":"literal","value":":","description":"\":\""}],"found":"}","offset":9,"line":1,"column":10,"name":"SyntaxError"},"message$":"Expected \":\" but \"}\" found.","plugin":{}},"callpoint":""},"duration":13,"level":"error","plugin_name":"transport","plugin_tag":"-","pattern":"cmd:listen,role:transport","when":1482487829552}

running a very simple service of mine
const seneca = require('seneca')()

seneca.add({cmd: 'salestax'}, function (msg, done) {
  var rate  = 0.23
  var total = msg.net * (1 + rate)
  console.log('Calculating ', msg.net)
  done(null, {total: total})
})

seneca.use('mesh', {
  isbase: true,
  pin: 'salestax'
})

I fixed the above error by changing my pin to cmd:salestax
However I'm still unable to get it working even with the examples/multicast:
my client says:
```

This is fixed


dea_next failing pre-packaging

Running "bosh create release" locally then uploading. I get the following error
Building dea_next...
  No artifact found for dea_next
  Generating...
  Pre-packaging...
  > + set -e -x
  > + cd /tmp/d20160921-10354-1l5wott/d20160921-10354-er3zaz/dea_next
  > + BUNDLE_WITHOUT=development:test
  > + bundle package --all --no-install --path ./vendor/cache
  > Fetching gem metadata from https://rubygems.org/
  > Fetching version metadata from https://rubygems.org/
  > Fetching https://github.com/cloudfoundry/warden.git
  > Fetching https://github.com/intridea/grape.git
  > Fetching https://github.com/cloudfoundry/loggregator_emitter.git
  > Fetching https://github.com/nats-io/ruby-nats.git
  > There was a RuntimeError while loading nats.gemspec:
  > can't modify frozen String from
  >   /tmp/d20160921-10354-1l5wott/d20160921-10354-er3zaz/dea_next/vendor/cache/ruby/1.9.1/bundler/gems/ruby-nats-3a3999d40c0b/nats.gemspec:10:in `block in <main>'
'dea_next' pre-packaging failed

Steps leading up to this were:
git clone https://github.com/cloudfoundry/cf-release.git
cd cf-release
git submodule update --init --recursive # because acceptance tests was empty
bosh create release

Note: In favor of doing this accurately I also did
git clone --recursive https://github.com/cloudfoundry/cf-release.git cf-release2
cd cf-release2
bosh create release

with the same results as above
Building dea_next...
  No artifact found for dea_next
  Generating...
  Pre-packaging...
  > + set -e -x
  > + cd /tmp/d20160921-25594-1nuxs2u/d20160921-25594-1c88kay/dea_next
  > + BUNDLE_WITHOUT=development:test
  > + bundle package --all --no-install --path ./vendor/cache
  > Fetching gem metadata from https://rubygems.org/
  > Fetching version metadata from https://rubygems.org/
  > Fetching https://github.com/cloudfoundry/warden.git
  > Fetching https://github.com/intridea/grape.git
  > Fetching https://github.com/cloudfoundry/loggregator_emitter.git
  > Fetching https://github.com/nats-io/ruby-nats.git
  > There was a RuntimeError while loading nats.gemspec:
  > can't modify frozen String from
  >   /tmp/d20160921-25594-1nuxs2u/d20160921-25594-1c88kay/dea_next/vendor/cache/ruby/1.9.1/bundler/gems/ruby-nats-3a3999d40c0b/nats.gemspec:10:in `block in <main>'
'dea_next' pre-packaging failed

I did notice that this is using ruby 1.9.3 so opted to try giving 2.3.1 a shot.
I repeated the above steps with ruby 2.3.1 and it appears to be still using 1.9.1
Building dea_next...
  No artifact found for dea_next
  Generating...
  Pre-packaging...
  > + set -e -x
  > + cd /tmp/d20160921-31694-1p0rwi6/d20160921-31694-pmvzz3/dea_next
  > + BUNDLE_WITHOUT=development:test
  > + bundle package --all --no-install --path ./vendor/cache
  > Fetching gem metadata from https://rubygems.org/
  > Fetching version metadata from https://rubygems.org/
  > Fetching https://github.com/cloudfoundry/warden.git
  > Fetching https://github.com/intridea/grape.git
  > Fetching https://github.com/cloudfoundry/loggregator_emitter.git
  > Fetching https://github.com/nats-io/ruby-nats.git
  > There was a RuntimeError while loading nats.gemspec:
  > can't modify frozen String from
  >   /tmp/d20160921-31694-1p0rwi6/d20160921-31694-pmvzz3/dea_next/vendor/cache/ruby/1.9.1/bundler/gems/ruby-nats-3a3999d40c0b/nats.gemspec:10:in `block in <main>'
'dea_next' pre-packaging failed

Line 10 from nats.gemspec
     1
     2
     3	lib = File.expand_path('../lib/', __FILE__)
     4	$:.unshift lib unless $:.include?(lib)
     5
     6	require 'nats/version'
     7
     8	spec = Gem::Specification.new do |s|
     9	  s.name = 'nats'
    10	  s.version = NATS::VERSION

lib/nats/version.rb
     1	module NATS
     2	  VERSION = "0.8.0".freeze
     3	end

Any ideas?
I don't see where it's trying to modify the version string.

Based on https://github.com/bundler/bundler/issues/3187 it looks like this may be an issue with ruby 1.9.3. I've uninstalled bosh which was using 1.9.3 and installed it with 2.3.1. I re-ran bosh after installing hashie and minitest and it build a working version. I will open an issue with bosh to require a min version of ruby.


Mix resolution dataset on Mip Nerf 360 dataset

Hi,
I have a question about your mix resolution dataset as described in the paper. From my understanding, you guys downscale the image to 4X and then up-scale it again to the original resolution. After than, you use colmap or Nerfstudio to get the camera pose. Would you mind sharing that dataset since the link in the readme is from the original Mip-nerf 360.

Hi thanks for your interest in our work. Sure, you can download the images and sfm from here.


Investigate threading of resource loading

There is potential for load improvements by threading/goroutining material loading. Most materials are referenced ahead of world loading in the material dictionary. In theory these could be loaded simultaneously first, instead of consecutively as they are now

Performance is currently reasonable, and code complexity is low.
Closing for now, it may be revisited sometime in the future


Login method not working

Same as reported by someone else.
Login method doesn't seem to work, error message when calling the isBanned method.
Fatal error: Uncaught seregazhuk\PinterestBot\Exceptions\AuthRequired: Error calling seregazhuk\PinterestBot\Api\Providers\User::isBanned method. You must log in before
Tried:
var_dump($bot->getLastError());
output
NULL
Tried the credentials in the browser, working fine there. Account not blocked etc.

same here, account seem not blocked, still getting traffic but PHP bot didn't work, maybe an new update from Pinterest?

Same problem here. I can log in the browser with the same credentials used in the bot.

Yes, looks like Pinterest has changed its API 😞

Sergey,
Do you think the login issue can be resolved?
Similar to other users, my sign in credentials work in the browser.

Do you think the login issue can be resolved?

Not now. Currently, I don't have free time to fix it 😞

Would be happy to help with resolving it, do you already have an idea at what needs to be changed where? Pinterest docs only talk about oAuth with a registered app, so that's not really helpful.
The URL you're using to create the user session
https://www.pinterest.co.uk/resource/UserSessionResource/create/
does work in the browser (based on cookies I also get my user info back in the browser, so the endpoint itself is working).

any solution ?

any solution ? help!

Didn't get much further with my digging, but maybe you did?!

Yes, I did. And they have changed the whole authentication system. It is not a matter of one request. So feel free to provide PR to fix it.

I am happy to help....., just need a little bit of guidance to get  started...
Is there anywhere I can see the authentication system?

Didn't get much further with my digging, but maybe you did?!

Yes, I did. And they have changed the whole authentication system. It is not a matter of one request. So feel free to provide PR to fix it.

when I try to use Postman for post request, Pinterest returns always the page "need to activate Javascript".
But in the Cookie we can find "csrftoken"

I am happy to help....., just need a little bit of guidance to get started...
Is there anywhere I can see the authentication system?
In my use of the bot, I really only require logging in to get pin activity.
All other parts of the bot, as far as I can see, work very well.

use chrome extension like Web Sniffer to trace login process and find the variables and the way that use pintreset to login
for debugging the request of this package edit ( C:...\vendor\seregazhuk\pinterest-bot\src\Api\CurlHttpClient.php ) and add before the return line of [function init($url, $postString, $headers)] this lines
curl_setopt($this->curl, CURLOPT_VERBOSE, true);
$verbose = fopen('c:\temp.txt', 'a+');
curl_setopt($this->curl, CURLOPT_STDERR, $verbose);

for registration of new account it work and the response of bot->register() give auth=1 but i suggest to save the cookies response _pinterest_sess and csrftoken to autologin (to prevent pinterest ban)

for the login still not working

I am happy to help....., just need a little bit of guidance to get started...
Is there anywhere I can see the authentication system?
In my use of the bot, I really only require logging in to get pin activity.
All other parts of the bot, as far as I can see, work very well.

use chrome extension like Web Sniffer to trace login process and find the variables and the way that use pintreset to login
for debugging the request of this package edit ( C:...\vendor\seregazhuk\pinterest-bot\src\Api\CurlHttpClient.php ) and add before the return line of [function init($url, $postString, $headers)] this lines
curl_setopt($this->curl, CURLOPT_VERBOSE, true);
$verbose = fopen('c:\temp.txt', 'a+');
curl_setopt($this->curl, CURLOPT_STDERR, $verbose);
for @seregazhuk

for registration of new account it work and the response of bot->register() give auth=1 but i suggest to save the cookies response _pinterest_sess and csrftoken to autologin (to prevent pinterest ban) at Auth.php
``
/**

@param Registration $registrationForm

@return bool|mixed
*/
protected function makeRegisterCall(Registration $registrationForm)
{
if (!$this->sendEmailVerificationAction()) {
return false;
}

$data = $registrationForm->getData();
$this->request->loadCookiesFor($data['email']);
<<< line to add
if (!$this->post(UrlBuilder::RESOURCE_CREATE_REGISTER, $registrationForm->toArray())) {
return false;
}
if (!$this->sendRegistrationActions()) {
return false;
}
return $this->completeRegistration();
}

``

i change also the URL_BASE and HOST at  \vendor\seregazhuk\pinterest-bot\src\Helpers\UrlBuilder.php i notice that pinterest not share correctly the cookies between subdomain
const URL_BASE = 'https://www.pinterest.com/';
const HOST = 'www.pinterest.com';
for the login still not working  and an alternative and tomporary solution is using bot->register() to login like edited in (1)

this didnt work for me... call $bot->auth->register() function with existing account will return false

Thanks for the guidance @tuxado ...
I cannot get $bot->auth->register() to work for an existing account.
I dont know if its relevant, but when logging in through Pinterest it sends requests to

Hi, Now it work only when you register new account. i create a fork to explain you how you can edit the vendor at https://github.com/tuxado/php-pinterest-bot
foreach action ContextLogResource, ActiveUserResource, EmailExistsResource and so ...... create a Trait file on php-pinterest-bot/src/Api/Traits/ like for example the EmailExistsResource that i created and before the login process you can call each request action from the Trait.
Note some actions aren't important for the login process. Also pinterest check if the request come from a real browser and from the same Orgin of The Host it why you get 404 it is why i use www.pinterest.com and not uk.pinterest.com but note also that the www.pinterest.com sometimes it can return some errors

I look like the login rework using this update https://github.com/tuxado/php-pinterest-bot
.

[INFO] Account<EMAIL_ADDRESS>[INFO] Account login success

waiting confirmation

Error calling seregazhuk\PinterestBot\Api\Providers\User::isBanned method. You must log in before
login fail and return this error plz anyone can help me thanks advance  @seregazhuk

also your examples is not working @seregazhuk

@anasnawaz use this package https://github.com/tuxado/php-pinterest-bot download it manually from the browser unzip it and copy and replace the main package on vendor/seregazhuk/php-pinterest
'
$bot = PinterestBot::create();
<EMAIL_ADDRESS>"password");
if($bot->auth->isLoggedIn())
{
if (!$bot->user->isBanned())
{
echo ' > [INFO] Account login success';
}
}

bundle of thanks  for help
On Tue, 12 Feb 2019 at 21:33, tuxado<EMAIL_ADDRESS>wrote:

@anasnawaz https://github.com/anasnawaz use this package
https://github.com/tuxado/php-pinterest-bot download it manually from the
browser unzip it and copy and replace the main package on
vendor/seregazhuk/php-pinterest
'
$bot = PinterestBot::create();
<EMAIL_ADDRESS>"password");
if($bot->auth->isLoggedIn())
{
if (!$bot->user->isBanned())
{
echo ' > [INFO] Account login success';
}
}
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/seregazhuk/php-pinterest-bot/issues/442#issuecomment-462831475,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AizzzO-AETdf1seC5PxUwnfhPr08C8Prks5vMuy-gaJpZM4afvmK
.

Thanks @tuxado
The new fork does achieve login and I'm now able to run $bot->pins->activity($pinid,25) ->toArray();
which is login dependent.

Thank you @tuxado
Actually, the issue was only with the domain. You should use www.pinterest.com instead of country-specific uk.pintrerest.com 🤦♂
Everybody, I have published a new release v5.9.1 with fixes. Please, update you versions.

Thanks @seregazhuk

@seregazhuk Thanks 👍

when my script loads in the browser there is an error "Sorry! We've had to block actions (logins) from your area because of suspicious activity. Please try again later." How can I handle it?

Thank you @tuxado
Actually, the issue was only with the domain. You should use www.pinterest.com instead of country-specific uk.pintrerest.com 🤦♂
Everybody, I have published a new release v5.9.1 with fixes. Please, update you versions.

Hello,
How can I update php-pinterest-bot v5.9.1?
when I run "composer require seregazhuk/pinterest-bot" in cmd, it loads "Installing
seregazhuk/pinterest-bot (5.7.3): Loading from cache"
BR

Thank you @tuxado
Actually, the issue was only with the domain. You should use www.pinterest.com instead of country-specific uk.pintrerest.com man_facepalming
Everybody, I have published a new release v5.9.1 with fixes. Please, update you versions.

Hello,
How can I update php-pinterest-bot v5.9.1?
when I run "composer require seregazhuk/pinterest-bot" in cmd, it loads "Installing
seregazhuk/pinterest-bot (5.7.3): Loading from cache"
BR

edit your composer.json
"seregazhuk/pinterest-bot": "^5.9.1",
and open CMD and type : composer update
or use require
composer require vendor/package:version
composer require seregazhuk/pinterest-bot:5.9.1
or
composer require vendor/package "version"
composer require seregazhuk/pinterest-bot "^5.9.1"

started to observe this issue now with v5.9.1 :(

I'm having the same problems again :(
Please help immediately.

I'm having the same problems again :(
Please help @seregazhuk  @tuxado

v5.9.1 doesn't work on me either. I keep getting the following error. friends please help me. @seregazhuk @tuxado
... Error calling seregazhuk\PinterestBot\Api\Providers\User::isBanned method. You must log in before. in ...

Thank you @tuxado
Actually, the issue was only with the domain. You should use www.pinterest.com instead of country-specific uk.pintrerest.com man_facepalming
Everybody, I have published a new release v5.9.1 with fixes. Please, update you versions.

Hello,
How can I update php-pinterest-bot v5.9.1?
when I run "composer require seregazhuk/pinterest-bot" in cmd, it loads "Installing
seregazhuk/pinterest-bot (5.7.3): Loading from cache"
BR

edit your composer.json
"seregazhuk/pinterest-bot": "^5.9.1",
and open CMD and type : composer update
or use require
composer require vendor/package:version
composer require seregazhuk/pinterest-bot:5.9.1
or
composer require vendor/package "version"
composer require seregazhuk/pinterest-bot "^5.9.1"

Iz there any update?


Resolve Update the HttpClient ,TcpClient ,Tcpserver,Operation client and OperationServer Services

Fixes #197

Reviewer is @DanaSunal


Failure

There was a problem with your controller program.
Look at the action logs for more details.

@omichel,
I was able to run the controller on my PC successfully. I'm posting here the logs FYI:


No error thrown when attempting to ban a moderator

Issue Description
When banning a user with subreddit.banned.add(), if the user is a moderator they will be unable to be banned, as you cannot ban a moderator. However, no error is thrown, despite the action failing.
System Information

PRAW Version: 5.3.0
Python Version: 2.7.14
Operating System: Ubuntu 17.10 x86_64

That's strange - when I try to reproduce this, I get the praw.exceptions.APIException CANT_RESTRICT_MODERATOR, like it should be.

Confirmed. I also receive:
APIException: CANT_RESTRICT_MODERATOR: 'You can't perform that action because that user is a moderator.' on field 'name'

@DragonBallZeke perhaps there was a hiccup on Reddit's end, but it seems as if all is good now. Please reopen with more information is you can still reproduce.


API-1843: KMSv2 Feature Gate

Feature: KMS Encryption Provider for sensitive etcd Resources
A user-configurable interface to support encryption of data stored in etcd using a supported Key Management Service (KMS).
OpenShift would need to align closer with KMS evolution upstream with respect to the different Kubernetes Encryption Providers available today that can encrypt resources from APIServer EncryptionConfig, https://kubernetes.io/docs/tasks/administer-cluster/kms-provider/.
User Stories:

As an OpenShift administrator, I want to encrypt secrets in my cluster at rest using KMS keys so that I can comply with my organization's security requirements.
As an OpenShift administrator, I want to let the KMS provider to manage the lifecycle of the encryption keys.

xref: OCPSTRAT-108, API-1684

/test all

/test all

/test all

/retest

Please add an appropriate PR description to explain the motivation behind this change. Neither the PR, nor the card linked really explain what we are doing here, or why

@JoelSpeed added description about the feature.
also, for context: #2035
Not sure though why some of the e2e(s) are still failing.

/retest

/retest

According to the upstream documentation, this feature is stable as of 1.29, which would be 4.16.
It is my understanding that if an upstream feature is declared stable, and we have not otherwise specified the gate status, then the feature is already enabled.
This PR appears to disable a feature that has previously been enabled.
/hold

/test ci/prow/verify

/test verify

This PR appears to disable a feature that has previously been enabled.

Yes, that was the root cause causing those e2e failures err="cannot set feature gate KMSv2 to false, feature is locked to true". KMSv2 feature gate (https://github.com/kubernetes/enhancements/issues/3299) is enabled by default upstream and even HyperShift uses it today so we do not have much gating to do on the upstream side. Today, in OpenShift we do not have the supported encryption providers to use the KMS v2 behaviour in downstream. Renaming the feature gate to KMSEncryptionProvider so we could gate our downstream implementation of the EncryptionConfig(s) through TP.
Thanks @JoelSpeed!
xref: https://redhat-internal.slack.com/archives/C06DXMRLXQT/p1729766294150549

PR is good to go, but we should get the EP into a state where the first round of reviews are in before we merge the gate, if we don't have at least some consensus, merging this would add a gate that is unneeded

/test all

/retest

thanks @tkashem
/remove-hold
off to you @JoelSpeed

/label acknowledge-critical-fixes-only

/retest
Looks good, but I'd like to see if these test failures are related

/lgtm
PR is inert, I don't believe the tests failures are related and relevant components (eg KAS) don't recognise the gate, so we've got past my concern about overlapping gates

/hold It sounds like this feature might need a little more time to bake, I'd like to hold this until the API is ready to go and the EP has at least had some review.
From discussion with @wallylewis we don't need to rush this into 4.18 so lets wait until after branch cut

/hold

/retest

/lgtm
/hold cancel


Unclear definition hub wins or member wins

I didn't success to create a 'pull requests' in this repo, but I would like to share my opinion regarding this documentation.   In the section 'Create a new Sync Group' , step In the Conflict Resolution section, select "Hub wins" or "Member wins." would prefer to add the definition about Hub and member. And what is ' Hub wins'  and ' Member wins' .
I would like to recommend to write like below:
In the Conflict Resolution section, select "Hub wins" or "Member wins." Hub is the database which you selected,  and the member is the database that will take part in the synchronization the specified column or table in the same sync group. When conflicts occurred,  the choice of this section will decide which of them will act as the obedience.
I understood how does data sync working when I finish my first reading of this article. Even though my sentences may not be the best descriptions, but it will help readers understand better when they are in each step. Thank you :)
Azure documentation issue guidance
Thanks for opening an issue in the Azure technical documentation repository.
We use GitHub issues as the primary channel for customer and community feedback about the Azure documentation.
Creating an issue
We prefer that you create documentation feedback issues using the Feedback link on the published article - the feedback control on the doc page creates an issue that contains all the article details so you can focus on the feedback part.
You can also create a feedback issue here in the repo. If you do this, please make sure your issue lists:

[ ] The relevant Azure service or technology.
[ ] A link to the published documentation article that you have feedback about.
[ ] Clear, specific feedback that the author can act on.

Pull requests and article contributions
If you know the change that is needed in an article, we encourage you to submit the changes directly using a pull request. If the change is large, or if you want to contribute an entire article, follow these guidelines:

[ ] Don't surprise us with a big pull request or a pull request with a new article! Submit an issue that describes the details of the proposed large change or new article.
[ ] Include the service or technology area.

We'll route the issue to the appropriate content team for review and discussion.
Tech support and product feedback
If you would like to contact Microsoft about other things, such as product feedback or tech support, please review these guidelines:

If you need technical support using Azure, the paid and free support options are described here: https://azure.microsoft.com/support/options/.

Each article in the Azure technical documentation contains a product feedback button - it's best to submit product feedback directly from a relevant article. Otherwise, you can submit product feedback for most Azure products in the following product feedback forum: https://feedback.azure.com/forums/34192--general-feedback.

@cloudmelon Can you please provide us with the doc URL for the doc in question?

H Bryan, here is the link :
https://docs.microsoft.com/en-us/azure/sql-database/sql-database-get-started-sql-data-sync
It means we can't do pull request to work directly on azure documentation
or I was just not found the right way to do that  ?
On Tue, Mar 27, 2018 at 7:41 PM, Bryan Trach<EMAIL_ADDRESS>wrote:

@cloudmelon https://github.com/cloudmelon Can you please provide us
with the doc URL for the doc in question?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/MicrosoftDocs/azure-docs/issues/6352#issuecomment-376612326,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEaE-Efb6hgn8Z5JdPg5J6SYY0ONbkvlks5tinm7gaJpZM4S9IYd
.

@cloudmelon Thanks for the feedback! I have assigned the issue to the content author to investigate further and update the document as appropriate.
The Azure community does have the ability to edit documents. Please see the following link for guidance: https://github.com/MicrosoftDocs/azure-docs

@cloudmelon there is an edit button in the upper right-hand corner of each Azure topic that enables you to submit your edit for review.

@cloudmelon Thank you for your suggestions. To clarify the role of the Hub Database, I am adding this sentence to the page: "The hub database is the central endpoint of the sync topology, in which a sync group has multiple database endpoints. All other database endpoints in the same sync group - that is, all member databases - sync with the hub database."

@cloudmelon To clarify the conflict resolution, I am adding this paragraph to the page:
"Hub wins" means that, when a conflict occurs, the data in the hub database overwrites the conflicting data in the member database. "Member wins" means that, when a conflict occurs, the data in the member database overwrites the conflicting data in the hub database.

I think this removes the lack of clarity. Let me know what you think.

@cloudmelon FYI the updated page will be published this afternoon (Monday, April 2, Redmond time).

Thank you :)
Melony QIN

On 2 Apr 2018, at 20:16, Douglas Laudenschlager<EMAIL_ADDRESS>wrote:
@cloudmelon FYI the updated page will be published this afternoon (Monday, April 2, Redmond time).
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

#please-close


Improve error message when trying to retrieve an artifact which doesn't exist

Fixed in https://github.com/simvue-io/client/pull/166


LongLora suport

⚠ Please check that this feature request hasn't been suggested before.

[X] I searched previous Ideas in Discussions didn't find any similar feature requests.
[X] I searched previous Issues didn't find any similar feature requests.

🔖 Feature description
Can you implement this new LoRA method? That would be great to have 32k+ LoRA models. Looks promising.
✔ Solution
https://github.com/dvlab-research/LongLoRA
http://arxiv.org/abs/2309.12307
❓ Alternatives
No response
📝 Additional Context
No response
Acknowledgements

[X] My issue title is concise, descriptive, and in title casing.
[X] I have searched the existing issues to make sure this feature has not been requested yet.
[X] I have provided enough information for the maintainers to understand and evaluate this request.

Seems like they provided a patch for llama in their repo.
Parts I've noticed:

during merge: need to load this

    trainable_params = os.path.join(args.peft_model, "trainable_params.bin")
    if os.path.isfile(trainable_params):
        model.load_state_dict(torch.load(trainable_params, map_location=model.device), strict=False)

for these layers "embed,norm"
https://github.com/dvlab-research/LongLoRA/blob/f34c8971b6c5a9cbd1e3a98d6483b750aef14cda/merge_lora_weights_and_save_hf_model.py#L98-L100

patch has a FA and non-FA version
https://github.com/dvlab-research/LongLoRA/blob/f34c8971b6c5a9cbd1e3a98d6483b750aef14cda/llama_attn_replace.py#L336

depend on rope_scaling also for base model
https://github.com/dvlab-research/LongLoRA/blob/f34c8971b6c5a9cbd1e3a98d6483b750aef14cda/fine-tune.py#L110-L113

May want to keep track of https://github.com/huggingface/peft/issues/958 in case it is supported there.

Is this something that is on the roadmap?


Window.set_visible segfault with gtk4_layer_shell

Bug description
Segfault only occurs after using gtk4_layer_shell::init_for_window to setup a
window. Using gtk::Window.set_visible then has a small chance to cause a
segfault. I've included a minimal repro to create a window, initialize it with
layer shell, and then toggle its visibility rapidly in order to make the
segfault occur faster.

Tested on compositors: Sway, Hyprland
Operating system: Arch Linux x86_64
Locally installed libraries:

gtk4 1:4.10.4-1
gtk4-layer-shell 1.0.0-1

Further details
For further details including repro code (rust) and logs, please see here:
https://github.com/gtk-rs/gtk4-rs/issues/1402

I've determined that this is not an issue with the rust bindings for gtk4 or layer shell (my original thought) but that it's either an issue with the gtk4-layer-shell or gtk4 libraries.
This repro in C also causes the segfault:
#include <gtk/gtk.h>
#include <gtk4-layer-shell/gtk4-layer-shell.h>
#include <stdio.h>

#define PROJECT_NAME "gtk4-layer-shell-segfault-repro-c"
#define TOGGLE_VISIBILITY_INTERVAL_MS 100

GtkWidget *window;
int count = 0;

static gboolean toggle_visible(gpointer user_data) {
  bool visible = !gtk_widget_get_visible(window);
  gtk_widget_set_visible(window, visible);
  count++;
  printf("<%d, %s>\n", count, visible ? "true" : "false");

  return true;
}

static void activate(GtkApplication *app, gpointer user_data) {
  window = gtk_application_window_new(app);

  printf("calling gtk_layer_init_for_window\n");
  gtk_layer_init_for_window(GTK_WINDOW(window));

  printf("toggling visibility rapidly\n");
  g_timeout_add(TOGGLE_VISIBILITY_INTERVAL_MS, toggle_visible, user_data);
}

int main(int argc, char **argv) {
  GtkApplication *app =
      gtk_application_new("lets.cause.a.segfault", G_APPLICATION_DEFAULT_FLAGS);
  g_signal_connect(app, "activate", G_CALLBACK(activate), NULL);

  int status = g_application_run(G_APPLICATION(app), argc, argv);
  g_object_unref(app);

  return status;
}

I've also filed this issue for GTK: https://gitlab.gnome.org/GNOME/gtk/-/issues/5881

Sorry for the delay. As I said in the other issue, probably best to close the issue on GTK as they can't help with gtk-layer-shell problems. This library does some weird stuff so any crash involving it should be assumed to be it's fault until proven otherwise.
I ran the example C code on Ubuntu 22.04 with various versions of GTK and libwayland built from source without being able to repro. Will try again in a couple weeks on Arch once I get home from my current trip.
In the mean time if someone who can repro can get it to crash in GDB and post a backtrace that would be helpful. Especially if you have debugging symbols for GTK and/or libwayland (either installed or are using a copy of those libraries built with them)

Sorry for the delay.

Don't worry about it. I really appreciate your hard work on gtk4-layer-shell and I'm happy that you're looking into this issue.
I've closed the issue on the GNOME gitlab instance. Hopefully I didn't waste too much of their time. I will try to be more considerate next time I have a bug to file. As for the repro, I'll try to see if I can test it in some more environments. I did not test it on GNOME, so it could be that it works fine on GNOME and crashes on other wayland compositors. If you didn't see it, I posted a GDB backtrace here. I'm not sure how to tell if this includes the symbols you mentioned. If this backtrace is insufficient, please let me know and I'll attempt to get another.
This isn't urgent for me, so please enjoy your trip.

Getting something similar:
free(): invalid pointer

Thread 1 "swaync" received signal SIGABRT, Aborted.
__pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0)
    at pthread_kill.c:44
44	     return INTERNAL_SYSCALL_ERROR_P (ret) ? INTERNAL_SYSCALL_ERRNO (ret) : 0;
--Type <RET> for more, q to quit, c to continue without paging--Quit
(gdb) bt
#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0)
    at pthread_kill.c:44
#1  0x00007ffff6c688b3 in __pthread_kill_internal (signo=6, threadid=<optimized out>) at pthread_kill.c:78
#2  0x00007ffff6c17abe in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3  0x00007ffff6c0087f in __GI_abort () at abort.c:79
#4  0x00007ffff6c0160f in __libc_message (fmt=fmt@entry=0x7ffff6d7b52f "%s\n")
    at ../sysdeps/posix/libc_fatal.c:150
#5  0x00007ffff6c72775 in malloc_printerr (str=str@entry=0x7ffff6d78fcc "free(): invalid pointer")
    at malloc.c:5651
#6  0x00007ffff6c74654 in _int_free (av=<optimized out>, p=p@entry=0xb50ec0, have_lock=have_lock@entry=0)
    at malloc.c:4425
#7  0x00007ffff6c771ce in __GI___libc_free (mem=mem@entry=0xb50ed0) at malloc.c:3367
#8  0x00007ffff7c87895 in g_free (mem=0xb50ed0) at ../glib/gmem.c:232
#9  0x00007ffff7c816a9 in g_source_unref_internal (source=0xc7ca60, context=0x50a550, have_lock=<optimized out>)
    at ../glib/gmain.c:2400
#10 0x00007ffff7c834d8 in g_main_dispatch (context=0x50a550) at ../glib/gmain.c:3494
#11 g_main_context_dispatch (context=0x50a550) at ../glib/gmain.c:4200
#12 0x00007ffff7ce1438 in g_main_context_iterate.isra.0
    (context=0x50a550, block=1, dispatch=1, self=<optimized out>) at ../glib/gmain.c:4276
#13 0x00007ffff7c80a23 in g_main_context_iteration (context=context@entry=0x50a550, may_block=may_block@entry=1)
    at ../glib/gmain.c:4343
#14 0x00007ffff7eb7ffd in g_application_run (application=0x66c910, argc=<optimized out>, argv=0x0)
    at ../gio/gapplication.c:2573
#15 0x00000000004116eb in sway_notification_center_main (args=0x7fffffffd158, args_length1=7)
    at ../src/main.vala:82
#16 0x0000000000411725 in main (argc=7, argv=0x7fffffffd158) at ../src/main.vala:13

Finally back to this. Unable to reproduce with the C example code.
Distro: Arch
gtk-layer-shell: current main (same commit as 1.0.1)
gtk: 4.12.1
compositors:

hyprland: 0.29.0
sway 1.8.1

I've let it run for at least a thousand iterations on both sway and hyprland with no crash. Can you test again with up to date versions of everything to see if you can still repro?

I have the same issue.
In the next few days I'll try to create a simple example to replicate the issue.
(Arch Linux + Hyprland)

Ok, I'm able to replicate the issue with the following rust code
use gdk::prelude::ApplicationExt;
use gdk::prelude::ApplicationExtManual;
use gtk::traits::ButtonExt;
use gtk::{
    traits::{GtkWindowExt, WidgetExt},
    ApplicationWindow,
};

pub fn main() {
    let app = gtk::Application::builder().application_id("test").build();

    app.connect_startup(move |app| {
        let window = build_surface(app);

        let window2 = build_surface2(app);
        let label = gtk::Label::new(Some("Child panel"));
        window2.set_child(Some(&label));

        let button = gtk::Button::with_label("Open child panel");

        button.connect_clicked(move |_| {
            if window2.is_visible() {
                window2.hide();
            } else {
                window2.show();
            }
        });
        window.set_child(Some(&button));

        app.connect_activate(move |_| window.show());
    });

    app.run();
}

fn build_surface(app: &gtk::Application) -> ApplicationWindow {
    let window = gtk::ApplicationWindow::new(app);
    window.set_default_size(-1, 50);

    gtk4_layer_shell::init_for_window(&window);
    gtk4_layer_shell::set_layer(&window, gtk4_layer_shell::Layer::Overlay);

    gtk4_layer_shell::auto_exclusive_zone_enable(&window);

    gtk4_layer_shell::set_anchor(&window, gtk4_layer_shell::Edge::Left, true);
    gtk4_layer_shell::set_anchor(&window, gtk4_layer_shell::Edge::Top, true);
    gtk4_layer_shell::set_anchor(&window, gtk4_layer_shell::Edge::Right, true);

    window
}

fn build_surface2(app: &gtk::Application) -> ApplicationWindow {
    let window = gtk::ApplicationWindow::new(app);
    window.set_default_size(250, 250);

    gtk4_layer_shell::init_for_window(&window);
    gtk4_layer_shell::set_layer(&window, gtk4_layer_shell::Layer::Overlay);

    gtk4_layer_shell::auto_exclusive_zone_enable(&window);

    gtk4_layer_shell::set_anchor(&window, gtk4_layer_shell::Edge::Left, true);
    gtk4_layer_shell::set_anchor(&window, gtk4_layer_shell::Edge::Top, true);
    gtk4_layer_shell::set_anchor(&window, gtk4_layer_shell::Edge::Right, true);
    gtk4_layer_shell::set_anchor(&window, gtk4_layer_shell::Edge::Bottom, true);

    window
}

Gsk-CRITICAL **: 14:07:05.369: gsk_render
_node_unref: assertion 'GSK_IS_RENDER_NODE (node)' failed
[1]    261357 segmentation fault (core dumped)

I'm using arch linux and hyprland 0.29,

Update
One very good news one bad news
use gtk4::prelude::ApplicationExtManual;
use gtk4::ApplicationWindow;
use gtk4::prelude::WidgetExt;
use gtk4::prelude::GtkWindowExt;
use gtk4::Application;
use gtk4_layer_shell::LayerShell;
use gtk4::prelude::ApplicationExt;
use gtk4::prelude::ButtonExt;
use gtk4::Button;
use gtk4::Label;

pub fn main() {
    let app = Application::builder().application_id("test").build();

    app.connect_startup(move |app| {
        let window = build_surface(app);

        let window2 = build_surface2(app);
        let label = Label::new(Some("Child panel"));
        window2.set_child(Some(&label));

        let button = Button::with_label("Open child panel");

        button.connect_clicked(move |_| {
            if window2.is_visible() {
                window2.hide();
            } else {
                window2.show();
            }
        });
        window.set_child(Some(&button));

        app.connect_activate(move |_| window.show());
    });

    app.run();
}

fn build_surface(app: &Application) -> ApplicationWindow {
    let window = ApplicationWindow::new(app);
    window.set_default_size(-1, 50);

    window.init_layer_shell();
    window.auto_exclusive_zone_enable();

    window.set_anchor(gtk4_layer_shell::Edge::Left, true);
    window.set_anchor(gtk4_layer_shell::Edge::Top, true);
    window.set_anchor(gtk4_layer_shell::Edge::Right, true);

    window
}

fn build_surface2(app: &Application) -> ApplicationWindow {
    let window = ApplicationWindow::new(app);
    window.set_default_size(250, 250);

    window.init_layer_shell();
    window.auto_exclusive_zone_enable();

    window.set_anchor(gtk4_layer_shell::Edge::Left, true);
    window.set_anchor(gtk4_layer_shell::Edge::Top, true);
    window.set_anchor(gtk4_layer_shell::Edge::Right, true);
    window.set_anchor(gtk4_layer_shell::Edge::Bottom, true);

    window
}

Cargo.toml
gtk4-layer-shell = "0.2.0"
gtk4 = "0.7.3"
gdk4 = "0.7.3"

I changed the example code updating the gtk4-layer-shell crate to the last version, I changed the gtk4 and gdk4 to the latest version removing the features and package references and I have compiled gtk4-layer-shell from code.
Now the issue seems resolved but I don't know why (that's the bad news :smile:).

I've now released v1.0.2, which I suspect contains changes that fix this. If there's still a way to reproduce the crash with 1.0.2 feel free to re-open the issue. Please provide sample code, make files/cargo.toml files where applicable and versions of all relevant libraries (GTK, libwayland, and any language specific packages). I will look into it if I can reproduce.

Was running into this very issue. @wmww can confirm v1.0.2 fixes the issue. Thanks a lot.


AMPI_Ireduce only creates a request at the root

Original issue: https://charm.cs.illinois.edu/redmine/issues/1376

MPI_Ireduce and other "all-to-one" type nonblocking collectives currently only create and set the request on the root rank. The MPI standard says that all ranks should be able to wait on the resulting request, so we just need to create a pre-completed request elsewhere.

Original date: 2017-01-23 03:21:14

examples/ampi/creduce only waits on the result at the root. That should be changed as well.

Original date: 2017-02-12 02:09:31

Was the code not actually wrong, or did this get fixed? 'Rejected' without an explanation is rather hard to follow.

Original date: 2017-02-13 03:02:18

I don't remember marking this rejected? It is a real issue, so reopening...

Original date: 2017-02-13 19:53:56

~~https://charm.cs.illinois.edu/gerrit/#/c/2239/~~ https://github.com/UIUC-PPL/charm/commit/dad73e6fcbc71386959102cf3af16ff00b1a7be5


Improve assertions

I think there is a lack of assertions expressiveness using page objects:
// .ok() results with a very poor assertion message which is context-less.
// expected: true
// actual: false
assert.ok(page.title.isVisible, 'DocsHero');

// .equal() also leads to a non-contextual message
// expected: 'DocsHero'
// actual: 'NotAHero'
assert.equal(page.title.text, 'DocsHero');

I think we can get an inspiration from qunit-dom. It has very similar API:

isVisible
hasClass
hasText(contains in the page-object world)
lots...

What if we have own qunit extension like:
// expected: page title is visible
// actual: page title is not visible
assert.PO(page.title).isVisible(); // use of PO just for visibility

// expected: page title has text "DocsHero"
// actual: page title has text "NotAHero"
assert.PO(page.title).hasText('DocsHero');

assert.PO(page.title).hasAttribute('aria-role', 'button');

This way an assertion can get all the necessary information to provide the best message possible
Alternatives
Better integration with qunit-dom
There is also an option to connect page-object with qunit-dom. But page object doesn't provide a way to get a DOM element. If we even expose an element then assertion would become too verbose:
  assert.dom(page.title.element).hasText('DocsHero');

Also there will not be an ability to support custom(defined by user page.isOnTop) page object props.
ember-qunit-nice-errors
We can recommend to use "ember-qunit-nice-errors" for better error messages. AFAIK it instruments qunit assertion lines in the build time to provide an assertion source code as an assertion message.
So for the following test:
test('it works', function(assert) {
  assert.equal(page.title.text, 'DocHero');
});

you would get the following assetion message:
  assert.equal(page.title.text, 'DocHero');
  expected: 'DocHero'
  actual: 'NotAHero'

I feel like we still can improve readability of the tests code and assertion messages in runtime.

@ro0gr have you tried ember-qunit-nice-errors ? This other addon was my attempt to solve the same problem. It's qunit specific though.

@san650 thanks! Updated "Alternatives" section with "ember-qunit-nice-errors"

Alternative: Sinon offers a good assertion API which integrates into your test framework. Using this style API made it easy to offer good error messages but also keep the library frame agnostic (has hooks for QUnit, Mocha, Jasmine, etc.) I found this exceptionally useful in my own tests that use Sinon.
What this would look like
All page objects would have an assert namespace. Each assert would call the frameworks' global assert hook. For example a simple isVisible could look like this:
test('...', function() {
  page.assert.ok(page.title.isVisible);
})

Implementation
PageObject has an assert namespace with pass and fail methods. By default they are defined in the addon:
PageObject.assert.fail = function(msg) {
  throw new AssertionError(msg);
};
PageObject.assert.pass = function(msg) {
  // No-op
};

Then in a framework specific addon (i.e. ember-cli-page-object-qunit):
import QUnit from 'qunit';

PageObject.assert.fail = function(msg) {
  QUnit.assert.ok(false, msg);
};
PageObject.assert.pass = function(msg) {
  QUnit.assert.ok(true, msg);
};

The actual PageObject assertion API could look something like:
PageObject.assert.ok = function(descriptor) {
  let keys = [];
  let node = descriptor;
  do {
    keys.unshift(Ceibo.meta(node).key);
  } while (node = Ceibo.parent(node));
  let msg = `expected ${keys.join('.')} to be true`;
  if (descriptor) {
    PageObject.assert.pass(msg);
  } else {
    PageObject.assert.fail(msg);
  }
};

Maybe? Anyway take a look at how Sinon tackled this very same problem and I think their API solution would fit well in this use case.

@sukima I'm glad you also think assertions can be improved.
Regarding this snippet:
test('...', function() {
  page.assert.ok(page.title.isVisible);
})

page.title.isVisible returns a boolean so if we do like this we would be unable to get any context specific information:

which was the page node(page.title)
which property we assert against(isVisible)

I think to keep a sinon flavour it has to be re-written to something like:
test('...', function() {
  page.assert.isVisible(page.title);
})

which is pretty similar to the original proposal.

As mentioned in the Alternatives it is potentially possible to integrate with qunit-dom.
The  work on qunit asserionts improvements is going(slooowly) in the ember-qunit-page-object repo.
The idea is to provide API like:
  assert.po(component)
    .isVisible()
    .hasText('some text')
    not.contains('O_o'); // not implemented yet

Closing the issue cause it's totally possible to do it in a separate addon.


Add more IntoPyObject impls

This adds more IntoPyObject impls, mostly for reference types, plus a few that I missed in the first round. I generally tried to provide one where ever we also have a ToPyObject impl, but there could be still some that I missed somewhere. They will hopefully show up once I start migrating trait bounds.

How should we go about testing these? Probably some level of coverage will be reached once we start migrating off the older traits, though it would be nice to drive that coverage closer to 100% if possible.

I definitely agree here, we should strive to keep coverage high. Maybe we can try to implement the IntoPy<PyObject>/ToPyObject impls in terms of IntoPyObject impls, that would make sure that they're in sync and also cover them with the existing tests. What do you think of that.

Maybe we can try to implement the IntoPy/ToPyObject impls in terms of IntoPyObject impls, that would make sure that they're in sync and also cover them with the existing tests.

👍 I think that'd be a good way to reduce the amount of duplication, and then as we rotate out the old traits all that will be left behind is.
There might be a few cases where the old traits can't use these new ones (e.g. different bytes specialization behaviour maybe?), where I guess we just live with the duplication.

There might be a few cases where the old traits can't use these new ones (e.g. different bytes specialization behaviour maybe?), where I guess we just live with the duplication.

True, I guess all generic types can't do this, because of the bounds... Anyway doing it for "leaf" types will be better than not doing at all.

I looks like coverage has improved significantly from that. Now its pretty much just the generic collections (and for a lot of them just the reference) impls that are not covered. (I also found a few impls that I've missed previously, so I added those as well)


Stylus support

Any plans to include Stylus like u did for Elixir?

@nuernbergerA until there is an official api, you can install the stylus-loader and add something like
{
  test: /\.styl$/,
  loader: 'stylus-loader'
}

to your webpack.config.js, just as you would without Mix.

Things like that should manually added by you to your webpack.config.js file.


How to divide two columns in apache superset

Make sure these boxes are checked before submitting your issue - thank you!

[x ] I have checked the superset logs for python stacktraces and included it here as text if there are any.
[ x] I have reproduced the issue with at least the latest released version of superset.
[ x] I have checked the issue tracker for the same issue and I haven't found one similar.

Superset version
0.28.1
Expected results
I want to make a bar chart in apache superset that is based on relative numbers. To be more specific, I want to divide SUM(mængde i ton) / Indbyggertal, but this does not work. How do I do this?
Actual results
It is not working
Steps to reproduce

Which database engine is that? Different ones have different ways to quote things, could be SUM("something") or SUM(``something``)

Thanks, it worked


YAML anchors & key overrides

Is this feature request related to a new rule or cfn-lint capabilities?
rules
Describe the feature you'd like to request
I'm using YAML anchors like this:
Resources:
  Common:
    Type: AWS::CloudFormation::Stack
    Properties:
      TemplateURL: ./stack.yaml
      Parameters:
        &CommonParameters
        # Common parameters to be included in the stacks below
        Image: !Ref PG08Image
        # Specific parameters to be overridden in the stacks below
        Name: Common
  Web:
    Type: AWS::CloudFormation::Stack
    Properties:
      TemplateURL: ./stack.yaml
      Parameters:
        <<: *CommonParameters
        Name: Web

Which results in
E0000 Duplicate found "Name" (line 11)
template.yaml:11:9

Describe the solution you'd like
It would be great if this case (duplicate keys) is a separate rule and can be turned off. Turning of E0000 didn't work for me and I would probably not even want that.
Additional context
No response
Is this something that you'd be interested in working on?

[ ] 👋 I may be able to implement this feature request

Would this feature include a breaking change?

[ ] ⚠ This feature might incur a breaking change

However when I run the lint on the packaged template (aws cloudformation package), it passes fine as the initial values are omitted. So this is not such a huge deal, although having been able to lint the source files would be nice.

Going to think on this one.  One of the issues we have is since we are parsing into a python dict we lose the second key.  What this would probably look like is we continue to parse, raise the error and the final object.  I think this would be possible

@kddejong Any update on your thoughts here?
I would expect that cfn-lint would lint after YAML anchors are evaluated as this seems to be how most other CFN tools are working and the YAML anchor is a YAML paradigm which should be transparent to a YAML end-user consumer.
Anyways, interested in any updates you might have.

I did some testing of some various yaml parsing tools to hopefully help generate some solutions. Here is a snippet of the input template I used to test:
      Replicas:
        - &Replicas
          Region: {Ref: 'AWS::Region'}
          PointInTimeRecoverySpecification:
            PointInTimeRecoveryEnabled: {Ref: PointInTimeRecoveryEnabled}
          # [...]
        - Fn::If:
            - HasOhioReplica
            - <<: *Replicas
              Region: us-east-2
            - Ref: 'AWS::NoValue'

Processing with yq template.yaml gives the following... not very helpful to solving the issue.
      Replicas:
        - &Replicas
          Region: {Ref: 'AWS::Region'}
          PointInTimeRecoverySpecification:
            PointInTimeRecoveryEnabled: {Ref: PointInTimeRecoveryEnabled}
          # [...]
        - Fn::If:
            - HasOhioReplica
            - !!merge <<: *Replicas
              Region: us-east-2
            - Ref: 'AWS::NoValue'

Processing with PyYAML (python3 -c 'import yaml; print(yaml.dump(yaml.safe_load(open("template.yaml"))))') gives the following... which does seem to generate something cfn-lint may be happy with. Hopefully this helps generate some solutions.
      Replicas:
        - PointInTimeRecoverySpecification: &id002
            PointInTimeRecoveryEnabled:
              Ref: PointInTimeRecoveryEnabled
          Region:
            Ref: AWS::Region
          # [...]
        - Fn::If:
            - HasOhioReplica
            - GlobalSecondaryIndexes: *id001
              PointInTimeRecoverySpecification: *id002
              ReadProvisionedThroughputSettings: *id003
              Region: us-east-2
              Tags: *id004
            - Ref: AWS::NoValue


Rendering Code for Evaluation

Could you kindly provide the rendering code for evaluation?

Sorry for the late reply.
The readme.txt in test data says,
We capture the input data with a static camera set at an azimuth angle of 0°.
For evaluation, we generate four distinct views at azimuth angles of -75°, 15°, 105°, and 195°, respectively. (No modifications to elevation, radius, and other camera parameters.)

So, in training config, you could modify data.random_camera.n_val_views to 24, and set data.eval_timestamp to [], data.eval_pose_index to [1, 7, 13, 19] , and then run standard validation process to get evaluation renderings.
The above evaluation config setting is not tested because my code is on old machine and I cannot get access to it now. You can have a try and then compare the results with the rendering results provided by us.
If you meet any problem, feel free to let me know 😄

Thank you so much for your help!


Adding rules count

PR related to this: https://github.com/j-easy/easy-rules/issues/283

@Desislav-Petrov
Your PR LGTM, rebased and merged as 8748d602b26a444858e6ce85d7b4b246365a17d7. I just renamed the method to size to be consistent with the Set API and added you to the contributors list (see b888b85119db7adde3d9399c9d9bd1d0079731f4).
Thank you for your contribution!


(819) Reversal Invoice is submitted to Workday on a submission replaced by a correction

Some changes done, see if the naming is clearer now.
Other changes tomorrow.


Coredump running unit test with MKL Blas

Description
I build mxnet with MKL blas and MKLDNN. The build was successful, however, when I ran unit test, I got the following core dump. Note: I am using master branch so there is no local changes from me.
Error Message
test_mkldnn.test_convolution ... [23:12:00] ../src/executor/graph_executor.cc:2062: Subgraph backend MKLDNN is activated.
[23:12:00] ../src/executor/../operator/../common/utils.h:472:
Storage type fallback detected:
operator = Convolution
input storage types = [row_sparse, row_sparse, row_sparse, ]
output storage types = [default, ]
params = {"num_filter" : 4, "kernel" : (3,), "stride" : 2, }
context.dev_mask = cpu
The operator with default storage type will be dispatched for execution. You're seeing this warning message because the operator above is unable to process the given ndarrays with specified storage types, context and parameter. Temporary dense ndarrays are generated in order to execute the operator. This does not affect the correctness of the programme. You can set environment variable MXNET_STORAGE_FALLBACK_LOG_VERBOSE to 0 to suppress this warning.
[23:12:00] ../src/executor/../operator/../common/utils.h:472:
Storage type fallback detected:
operator = _backward_Convolution
input storage types = [default, row_sparse, row_sparse, row_sparse, ]
output storage types = [default, default, default, ]
params = {"num_filter" : 4, "kernel" : (3,), "stride" : 2, }
context.dev_mask = cpu
The operator with default storage type will be dispatched for execution. You're seeing this warning message because the operator above is unable to process the given ndarrays with specified storage types, context and parameter. Temporary dense ndarrays are generated in order to execute the operator. This does not affect the correctness of the programme. You can set environment variable MXNET_STORAGE_FALLBACK_LOG_VERBOSE to 0 to suppress this warning.
OMP: Error #15: Initializing libiomp5.so, but found libomp.so already initialized.
OMP: Hint This means that multiple copies of the OpenMP runtime have been linked into the program. That is dangerous, since it can degrade performance or cause incorrect results. The best thing to do is to ensure that only a single OpenMP runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library. As an unsafe, unsupported, undocumented workaround you can set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, but that may cause crashes or silently produce incorrect results. For more information, please see http://www.intel.com/software/products/support/.
Aborted (core dumped)
To Reproduce

rm -rf build
mkdir -p build && cd build
cmake -GNinja \
    -DUSE_CUDA=OFF \
    -DUSE_MKL_IF_AVAILABLE=ON \
    -DCMAKE_BUILD_TYPE=Release \
..
ninja

cd ..
pip install -e python

nosetests -v tests/python/mkl

Environment
We recommend using our script for collecting the diagnositc information. Run the following command and paste the outputs below:

----------Python Info----------
Version      : 3.6.6
Compiler     : GCC 7.2.0
Build        : ('default', 'Jun 28 2018 17:14:51')
Arch         : ('64bit', '')
------------Pip Info-----------
Version      : 19.3.1
Directory    : /home/ubuntu/.virtualenvs/mxnet/lib/python3.6/site-packages/pip
----------MXNet Info-----------
Version      : 1.6.0
Directory    : /home/ubuntu/src/mxnet/python/mxnet
Num GPUs     : 0
Hashtag not found. Not installed from pre-built package.
----------System Info----------
Platform     : Linux-4.4.0-1100-aws-x86_64-with-debian-stretch-sid
system       : Linux
node         : ip-172-31-45-221
release      : 4.4.0-1100-aws
version      : #111-Ubuntu SMP Wed Dec 4 12:20:15 UTC 2019
----------Hardware Info----------
machine      : x86_64
processor    : x86_64
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                36
On-line CPU(s) list:   0-35
Thread(s) per core:    2
Core(s) per socket:    18
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 85
Model name:            Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz
Stepping:              4
CPU MHz:               3000.000
BogoMIPS:              6000.00
Hypervisor vendor:     KVM
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              1024K
L3 cache:              25344K
NUMA node0 CPU(s):     0-35
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology nonstop_tsc aperfmperf tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single kaiser fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f rdseed adx smap clflushopt clwb avx512cd xsaveopt xsavec xgetbv1 ida arat pku
----------Network Test----------
Setting timeout: 10
Timing for MXNet: https://github.com/apache/incubator-mxnet, DNS: 0.0014 sec, LOAD: 0.5697 sec.
Timing for GluonNLP GitHub: https://github.com/dmlc/gluon-nlp, DNS: 0.0005 sec, LOAD: 0.7626 sec.
Timing for GluonNLP: http://gluon-nlp.mxnet.io, DNS: 0.1701 sec, LOAD: 0.6120 sec.
Timing for D2L: http://d2l.ai, DNS: 0.0159 sec, LOAD: 0.0476 sec.
Timing for D2L (zh-cn): http://zh.d2l.ai, DNS: 0.0120 sec, LOAD: 0.2423 sec.
Timing for FashionMNIST: https://repo.mxnet.io/gluon/dataset/fashion-mnist/train-labels-idx1-ubyte.gz, DNS: 0.0208 sec, LOAD: 0.5693 sec.
Timing for PYPI: https://pypi.python.org/pypi/pip, DNS: 0.0012 sec, LOAD: 0.4202 sec.
Timing for Conda: https://repo.continuum.io/pkgs/free/, DNS: 0.0092 sec, LOAD: 0.0551 sec.

export KMP_DUPLICATE_LIB_OK=TRUE works. Closing this issue.


Files protection: record files endpoint

Sub of https://github.com/inveniosoftware/invenio-app-rdm/issues/577
Deal with record files' endpoint: https://inveniordm.docs.cern.ch/reference/rest_api/#record-files

list
read file metadata
download (read file)

blocked until dealing with drafts elucidate all the changes needed in records-resources / drafts-resources. Then those need to be published.


Check for any unwanted effects from having multiple TOOL tokens in an entity definition

If not, may as well undo #85, checking for changes in the readme etc

They are removed! Time to undo this madness


Limits

Here I did two things:

change _default_limits to pyart names as a step to implement issue #43

change how display handle limits and colormap: remove limits (dict) and CMAP (string) and added Vlims and Vcmap both Variables holding dictionary.

Vlims keys: 'xmin' (float), 'xmax' (float), 'ymin' (float), 'ymax' (float)
Vcmap keys: 'vmin' (float), 'vmax' (float), 'cmap' (string)

this end up resulting in change throughout code, but while doing I really had the impression this way is much better, for instance as ZoomPan now work over Vlims this allow it to affect several display at the same time (just connect Vlims in ComponentControl)

A general comment, does the potential change in PyART issue #329 have an affect on this?  Should we wait for changes if so?

A small re-visitation may be needed when that issue is handle, however I see no reason to wait.

I mean, that issue is an enhancement, it won't break anything.

Thank you for merging the last alterations, I will use the current version of plot_radar to create plot_grid. I will try to avoid code duplication, but it will mostly probably have some

Sounds like a plan.
Just and FYI, I'm going to be working on getting some the exclude functions into ARTView over the next few weeks (instead of the mask I proposed in PyART issue). I haven't fully formed how I want to do this yet. It shouldn't affect any changes you make I think.

Ok, but please keep me informed as you progress on that. Also an FYI, see this commit of mine, this will give a more direct access to GateFilter so we can for instance built a filter from roi. I haven't open a pyart PR yet, just because I like to go by steps

Will do @gamaanderson that commit looks great - it will be very useful for ARTView.
I need to turn my focus on finishing a paper for the next couple of weeks, so expect some more work after that. In August, I'll ping the group for any last minute additions before the radar conference.


Audio Transcript: Error. Could not transcribe this audio.

Audio Transcript:
Error. Could not transcribe this audio.

Hi, can you provide more information about this?

from markitdown import MarkItDown
markitdown = MarkItDown()
result = markitdown.convert("******.mp3")
print(result.text_content)
.xlsx .pdf no problem,.mp3 format file error prompt
Audio Transcript:
Error. Could not transcribe this audio.

I am also getting issues with audio files. I tired with .mp3 and .wav file format and getting the below issue:

Additionally, I am using a Mac and noticed that exiftool was not installed during the pip installation. I’m unable to install this tool externally because macOS is blocking the installation and displaying a malware warning. Could someone assist with this? Your help would be greatly appreciated. Thank you!

I am also getting issues with audio files. I tried with .mp3 and .wav file format and getting the below issue:
Additionally, I am using a Mac and noticed that exiftool was not installed during the pip installation. I’m unable to install this tool externally because macOS is blocking the installation and displaying a malware warning. Could someone assist with this? Your help would be greatly appreciated. Thank you!

I am in window env,the reason for above problems is that I am unable to use google service.


Add CausalML Survey

Dear Ruocheng,
Thank you very much for maintaining this excellent list of Causality algorithms.
We recently published a survey on causal ML methods and thought it would fit well on this list.
Best wishes
Jean

Thanks, Jean!

I read some parts of your survey today, it is amazingly well written and very helpful for me :)

Thank you so much for adding it, your positive feedback and kind words, Ruocheng! It's so nice to hear that you find it useful!! 🥳


Use the exact given station name as friendly name

Is your feature request related to a problem? Please describe.
If I add a station and give it the name "DWD Home", it will create a weather sensor with the name: "weather.dwd_home_dwd_home". More importantly, the friendly_name attribute of the weather sensor will also be double the given name: "DWD Home DWD Home"
Describe the solution you'd like
The friendly name should be just "DWD Home" and the name "weather.dwd_home".
Additional context
The other sensors have names as they should, for example: "sensor.dwd_home_temperature".

Is implemented in 2.0.9


Change the getPool query result to PoolUnion

Return type getPool is PoolBase, but the data is prepared as PoolUnion. Might need to check if there is 100% overlap so it doesn't break things.
It's needed to read all data we produce in the query.

Not sure what you mean here. The data can be read using the fragment like
    ... on GqlPoolGyro {
      alpha
      beta
      type
      nestingType
      tokens {
        ... on GqlPoolToken {
          ...GqlPoolToken
        }
      }
    }


Add KLatexFormula.app v3.3.0beta

You could add an appcast
Example for sourceforge: https://github.com/caskroom/homebrew-cask/issues/16685

Thank you for the contribution. It needed some fixes, so they were made in 953ab6fe0793f155cf00ba7dd0a79a538359aa53. Your contribution is still included (and still credited to you), with the appropriate modifications. Please feel free to ask about any of the changes.


[BUG] Writing Alignment causes issues with positional metadata

I just hit upon a weird bug. To reproduce,
$ conda create -n alignment-bug python=2.7 scikit-bio=0.4.0
$ source activate alignment-bug

Then run the following script:
from StringIO import StringIO
import skbio as sk

seqs = [sk.DNA('ACGT', metadata={'id': 'foo'}),
		sk.DNA('AG-T', metadata={'id': 'bar'})]

aln = sk.Alignment(seqs)
print aln.position_counters()
aln.write('test.fa', format='fasta')
print aln.position_counters()

This will print (before the call to write()):
[Counter({'A': 2}), Counter({'C': 1, 'G': 1}), Counter({'-': 1, 'G': 1}), Counter({'T': 2})]

and then (after the call to write()):
Traceback (most recent call last):
  File "test_bug.py", line 11, in <module>
    print aln.position_counters()
  File "/home/das/anaconda/lib/python2.7/site-packages/skbio/alignment/_alignment.py", line 1227, in position_counters
    return [Counter(p) for p in self.iter_positions(constructor=str)]
  File "/home/das/anaconda/lib/python2.7/site-packages/skbio/alignment/_alignment.py", line 1041, in iter_positions
    position = [constructor(seq[i]) for seq in self]
  File "/home/das/anaconda/lib/python2.7/site-packages/skbio/sequence/_sequence.py", line 880, in __getitem__
    return self._to(sequence=seq, positional_metadata=positional_metadata)
  File "/home/das/anaconda/lib/python2.7/site-packages/skbio/sequence/_sequence.py", line 1969, in _to
    positional_metadata=positional_metadata)
  File "/home/das/anaconda/lib/python2.7/site-packages/skbio/sequence/_iupac_sequence.py", line 586, in _constructor
    return self.__class__(validate=False, lowercase=False, **kwargs)
  File "/home/das/anaconda/lib/python2.7/site-packages/skbio/sequence/_iupac_sequence.py", line 172, in __init__
    sequence, metadata, positional_metadata)
  File "/home/das/anaconda/lib/python2.7/site-packages/skbio/sequence/_sequence.py", line 585, in __init__
    self.positional_metadata = positional_metadata
  File "/home/das/anaconda/lib/python2.7/site-packages/skbio/sequence/_sequence.py", line 509, in positional_metadata
    (num_rows, len(self)))
ValueError: Number of positional metadata values (4) must match the number of characters in the sequence (1).

Note: Running the same code with version 0.2.3 works perfectly.
I realize that this may not be a concern since TabularMSA is taking over anyway. The only way I found to avoid the exception so far is to not write the alignment.
Cheers,
Dan

Thanks @dansondergaard this is a really weird one! One thing that caught my eye was the use of StringIO.StringIO instead of io.StringIO, but that doesn't seem to be the problem here.
If the problem is in Alignment then it probably isn't a blocker, but we should investigate to figure out why this is happening. Worst case, this is some strange behaviour from the sequence classes, in which case this is a blocking bug.
Until we can figure out why, let's assume the worst.

Tracked down the line that is causing it:
https://github.com/biocore/scikit-bio/blob/master/skbio/io/format/_base.py#L185
The check for positional metadata will end up creating it (but it will be empty). What is disconcerting is that the code should still work regardless of that fact.

It is a problem with Sequence:
import skbio

seq = skbio.DNA('ACGT', metadata={'id': 'foo'})
_ = seq.positional_metadata
seq[0]

also causes this error.

Got it! The logical error is that Sequence.__getitem__ checks has_positional_metadata which is False when it is both unset, and empty. By asking for positional_metadata we create an empty DataFrame on the object for the user to use. Because this is falsey, getitem doesn't try to pass on the sliced positional_metadata which means in the _to call existing positional metadata (with the wrong size) is inherited from the instance.

Nice sleuthing!

@ebolyen The StringIO should have been removed, that was my mistake. It's not used at all.
Really nice and quick debugging!


Sidebar component gives error because it is not adjcent to body tag

The error i'm getting is Had to move sidebar. For optimal performance make sure sidebar and pusher are direct children of your body tag
My layout render looks like this:
    return

      <Sidebar className='theSidebar' flux={flux} init={this.state.secondaryNav}><SecondaryNavigation /></Sidebar>
      <PrimaryNavigation flux={flux} />
      <Button className='icon' style={style.hamburger} onClick={this._expandNav.bind(this)}><Icon className='sidebar icon' color='black'></Icon></Button>
      <Map flux={flux} />
      <RouteHandler flux={flux}/>

;

My router render looks like this:
router.run(function(Handler) {
  React.render(<Handler />, document.body);
});

Of course putting the Sidebar element outside of the initial div doesn't work because it needs to be enclosed in a container element. So what is the solution to this?

For anyone else interested, I ended up finding out that I needed to change the context of my Sidebar element. This should probably be a required prop for this react component, as I don't see how anyone else would get it working otherwise.
var init = {context: '.mainContainer'};

return (

    <Sidebar className="theSidebar" init={init}><SecondaryNavigation /></Sidebar>

      <PrimaryNavigation />
      <Button className='icon' style={style.hamburger} onClick={this._expandNav}><Icon className='sidebar icon' color='black'></Icon></Button>
      <Map />
      <RouteHandler />

);

@michaelryancaputo Sorry for replying late.
You're right. On semanti-ui sidebar document, it should be set-up with that structure before initializing a sidebar.
ref. http://semantic-ui.com/modules/sidebar.html#/usage
BTW, what's the required prop mean? Use the specific structure for sidebar component?


changes to http2 test server

Not kill the server on disconnect
Spawn one server per test case

test this please

The failures seem unrelated. Merging.


Create the "build your own environment" quick start

This new topic follows the interactive quick starts. Readers will learn to install VS Code and .NET Core. They will validate the installation by creating a first console application.

I don't believe this should be a separate topic. The right action here is to point developers to the Get started in 10 minutes on the .NET Core site when they are ready to move beyond the inline interactive tutorials.
That's the quickest, and simplest way to get started. It would then point them back to the later quick starts that they would run on their machine.
/cc @rowanmiller
@rpetrusha Thoughts? Or is there a better page in docs for setting up your own environment?

I don't believe this should be a separate topic. The right action here is to point developers to the Get started in 10 minutes

That makes sense to me. We keep that up to date with every release, and do a bunch of dynamic stuff around OS detection etc.

@BillWagner and @rowanmiller, I agree. The Get Started in 10 minutes page is quite good, and there's really no need for us to reinvent the wheel.


fix(chat): fix new conversation loading spinner size for mobile (Issue #2701)

Description:
"New conversation" (Chat) loading spinner have incorrect size for mobile
Issues:

Issue #2701

Checklist:

[x] the pull request name complies with Conventional Commits
[x] the pull request name starts with fix(<scope>):, feat(<scope>):, feature(<scope>):, chore(<scope>):, hotfix(<scope>): or e2e(<scope>):. If contains breaking changes then the pull request name must start with fix(<scope>)!:, feat(<scope>)!:, feature(<scope>)!:, chore(<scope>)!:, hotfix(<scope>)!: or e2e(<scope>)!: where <scope> is name of affected project: chat, chat-e2e, overlay, shared, sandbox-overlay, etc.
[x] the pull request name ends with (Issue #<TICKET_ID>) (comma-separated list of issues)
[x] I confirm that do not share any confidential information like API keys or any other secrets and private URLs

/deploy-review