issue ?

i run all of them together so all of them should have same points
why the last 2 account have more points and all the rest have less ?
i think its something related to those refresh that bot is doing every second
============================================
|                 Teneo Bot                  |
|         github.com/recitativonika          |

================= Account 1 =================
Email<EMAIL_ADDRESS>User ID: 1
Points Total: 13775
Points Today: 1525
Message: Connected successfully
Proxy: 1:45001 (User: undefined)

================= Account 2 =================
Email<EMAIL_ADDRESS>User ID: 2
Points Total: 13675
Points Today: 1575
Message: Connected successfully
Proxy: 2:45001 (User: undefined)

================= Account 3 =================
Email<EMAIL_ADDRESS>User ID: 3
Points Total: 13350
Points Today: 1525
Message: Connected successfully
Proxy: 3:45001 (User: undefined)

================= Account 4 =================
Email<EMAIL_ADDRESS>User ID: 4
Points Total: 13475
Points Today: 1500
Message: Connected successfully
Proxy: 4:45001 (User: undefined)

================= Account 5 =================
Email<EMAIL_ADDRESS>User ID: 5
Points Total: 13550
Points Today: 1550
Message: Connected successfully
Proxy: 5:45001 (User: undefined)

================= Account 6 =================
Email<EMAIL_ADDRESS>User ID: 6
Points Total: 13475
Points Today: 1550
Message: Connected successfully
Proxy: 6:45001 (User: undefined)

================= Account 7 =================
Email<EMAIL_ADDRESS>User ID: 7
Points Total: 13375
Points Today: 1550
Message: Connected successfully
Proxy: 7:45001 (User: undefined)

================= Account 8 =================
Email<EMAIL_ADDRESS>User ID: 8
Points Total: 13550
Points Today: 1550
Message: Connected successfully
Proxy: 8:45001 (User: undefined)

================= Account 9 =================
Email<EMAIL_ADDRESS>User ID: 9
Points Total: 23650
Points Today: 2850
Message: Connected successfully
Proxy: 9:45001 (User: undefined)

================= Account 10 =================
Email<EMAIL_ADDRESS>User ID: 10
Points Total: 23650
Points Today: 2850
Message: Connected successfully
Proxy: 10:45001 (User: undefined)

Status:
Account 1: Potential Points: 9.39, Countdown: 9m 21s
Account 2: Potential Points: 8.54, Countdown: 9m 52s
Account 3: Potential Points: 9.39, Countdown: 9m 21s
Account 4: Potential Points: 8.55, Countdown: 9m 52s
Account 5: Potential Points: 8.55, Countdown: 9m 52s
Account 6: Potential Points: 8.63, Countdown: 9m 52s
Account 7: Potential Points: 9.4, Countdown: 9m 21s
Account 8: Potential Points: 9.39, Countdown: 9m 22s
Account 9: Potential Points: 8.54, Countdown: 9m 52s
Account 10: Potential Points: 10.35, Countdown: 9m 22s

Is your storage capacity increased?

Not really sure what cause that because I only run one account.  Maybe I will try figure it out sometime.
It should not because the refresh every second because it only refresh the time and the potential point.

Try to pull the repo again.

Is your storage capacity increased?

nope

Try to pull the repo again.

ok great thank you. i will try it now

Try to pull the repo again.

It's worse now :D
some of the accounts keep disconnecting and time goes back to 15m after couple of seconds

sorry about that,  for temporary solution you can download and use the old code for now. I will try fix it later.
https://github.com/recitativonika/teneo-bot/blob/65d88fe24122b41672a37a3861062bfae5ea5103/index.js

sorry about that, for temporary solution you can download and use the old code for now. I will try fix it later. https://github.com/recitativonika/teneo-bot/blob/65d88fe24122b41672a37a3861062bfae5ea5103/index.js

yeah i just reverse the git pull command so i am good now
thanks mate

sure, and thanks for reporting the problem. I can confirm the script randomly restart,  I think it because sometime their websocket have issue with the connection/connection dropped.
Will try add auto reconnect to the code instead restarting.

Please pull the repo again, I added BrowserID and change the websocket connection to auto-reconnect when disconnected or connection dropped. It should fix the point problem.
And I change how the proxy stored, now is in proxy.txt please edit it before using the new update.
Let me know if any problem still occured because I only test in 5 multi-account, will look into it later when I have time!

Please pull the repo again, I added BrowserID and change the websocket connection to auto-reconnect when disconnected or connection dropped. It should fix the point problem, not 100% because sometime the server just down for a moment but it should more consistent when sending ping/point. And I change how the proxy stored, now is in proxy.txt please edit it before using the new update.
Let me know if any problem still occured because I only test in 5 multi-account, will look into it later when I have time! You can use the old code with index-old.js. Just run node index-old.js if you want to run old code

cool i will try it right now
btw there is proxy,js and proxy.txt what should i use ?

ok i see that proxy.js is for the old one
i will use .txt

so after something like a minute its started one account
then after 3 minutes more 2 acounts
now after more 4 minutes no more account running (4 accounts on OLD)
Status:
Account 1: Potential Points: 0, Countdown: Calculating...
Account 2: Potential Points: 25, Countdown: Calculating, it might take a minute before starting...
Account 3: Potential Points: 0, Countdown: Calculating...
Account 4: Potential Points: 9.49, Countdown: 9m 18s
Account 5: Potential Points: 6.8, Countdown: 10m 55s
Account 6: Potential Points: 6.72, Countdown: 10m 58s
Account 7: Potential Points: 0, Countdown: Calculating...

so after something like a minute its started one account
then after 3 minutes more 2 acounts
now after more 4 minutes no more account running (4 accounts on OLD)
Status:
Account 1: Potential Points: 0, Countdown: Calculating...
Account 2: Potential Points: 25, Countdown: Calculating, it might take a minute before starting...
Account 3: Potential Points: 0, Countdown: Calculating...
Account 4: Potential Points: 9.49, Countdown: 9m 18s
Account 5: Potential Points: 6.8, Countdown: 10m 55s
Account 6: Potential Points: 6.72, Countdown: 10m 58s
Account 7: Potential Points: 0, Countdown: Calculating...

this errors was on the new code

my 5 account that i run for test (run about 2 day) it not having problem. Hmm.. I not sure what the cause of your problem. maybe try limit run 5 account?

my 5 account that i run for test (run about 2 day) it not having problem. Hmm.. I not sure what the cause of your problem. maybe try limit run 5 account?

i am running now with that index.js
const axios = require('axios');
const chalk = require('chalk');
const WebSocket = require('ws');
const { HttpsProxyAgent } = require('https-proxy-agent');
const readline = require('readline');
const accounts = require('./account.js');
const proxies = require('./proxy.js');
const { useProxy } = require('./config.js');
const rl = readline.createInterface({
input: process.stdin,
output: process.stdout
});
let sockets = [];
let pingIntervals = [];
let countdownIntervals = [];
let potentialPoints = [];
let countdowns = [];
let pointsTotals = [];
let pointsToday = [];
let lastUpdateds = [];
let messages = [];
let userIds = [];
const authorization = "Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJzdXBhYmFzZSIsInJlZiI6Imlra25uZ3JneHV4Z2pocGxicGV5Iiwicm9sZSI6ImFub24iLCJpYXQiOjE3MjU0MzgxNTAsImV4cCI6MjA0MTAxNDE1MH0.DRAvf8nH1ojnJBc3rD_Nw6t1AV8X_g6gmY_HByG2Mag";
const apikey = "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJzdXBhYmFzZSIsInJlZiI6Imlra25uZ3JneHV4Z2pocGxicGV5Iiwicm9sZSI6ImFub24iLCJpYXQiOjE3MjU0MzgxNTAsImV4cCI6MjA0MTAxNDE1MH0.DRAvf8nH1ojnJBc3rD_Nw6t1AV8X_g6gmY_HByG2Mag";
function displayHeader() {
console.log("");
console.log(chalk.yellow(" ============================================"));
console.log(chalk.yellow("|                 Teneo Bot                  |"));
console.log(chalk.yellow("|         github.com/recitativonika          |"));
console.log(chalk.yellow(" ============================================"));
console.log("");
console.log(chalk.cyan(_____________________________________________));
}
function displayAccountData(index) {
console.log(chalk.cyan(================= Account ${index + 1} =================));
console.log(chalk.whiteBright(Email: ${accounts[index].email}));
console.log(User ID: ${userIds[index]});
console.log(chalk.green(Points Total: ${pointsTotals[index]}));
console.log(chalk.green(Points Today: ${pointsToday[index]}));
console.log(chalk.whiteBright(Message: ${messages[index]}));
const proxy = proxies[index % proxies.length];
if (useProxy) {
console.log(chalk.hex('#FFA500')(Proxy: ${proxy.host}:${proxy.port} (User: ${proxy.username})));
}
console.log(chalk.cyan(_____________________________________________));
}
function logAllAccounts() {
console.clear();
displayHeader();
for (let i = 0; i < accounts.length; i++) {
displayAccountData(i);
}
console.log("\nStatus:");
for (let i = 0; i < accounts.length; i++) {
console.log(Account ${i + 1}: Potential Points: ${potentialPoints[i]}, Countdown: ${countdowns[i]});
}
}
async function connectWebSocket(index) {
if (sockets[index]) return;
const version = "v0.2";
const url = "wss://secure.ws.teneo.pro";
const wsUrl = ${url}/websocket?userId=${encodeURIComponent(userIds[index])}&version=${encodeURIComponent(version)};
const proxy = proxies[index % proxies.length];
const agent = useProxy ? new HttpsProxyAgent(http://${proxy.username}:${proxy.password}@${proxy.host}:${proxy.port}) : null;
sockets[index] = new WebSocket(wsUrl, { agent });
sockets[index].onopen = async () => {
lastUpdateds[index] = new Date().toISOString();
console.log(Account ${index + 1} Connected, lastUpdateds[index]);
startPinging(index);
startCountdownAndPoints(index);
};
sockets[index].onmessage = async (event) => {
const data = JSON.parse(event.data);
if (data.pointsTotal !== undefined && data.pointsToday !== undefined) {
lastUpdateds[index] = new Date().toISOString();
pointsTotals[index] = data.pointsTotal;
pointsToday[index] = data.pointsToday;
messages[index] = data.message;
logAllAccounts();

}
if (data.message === "Pulse from server") {
console.log(Pulse from server received for Account ${index + 1}. Start pinging...);
setTimeout(() => {
startPinging(index)
}, 10000);
}
};
sockets[index].onclose = () => {
sockets[index] = null;
console.log(Account ${index + 1} Disconnected);
restartAccountProcess(index);
};
sockets[index].onerror = (error) => {
console.error(WebSocket error for Account ${index + 1}:, error);
};
}
function disconnectWebSocket(index) {
if (sockets[index]) {
sockets[index].close();
sockets[index] = null;
restartAccountProcess(index);
}
}
function startPinging(index) {
pingIntervals[index] = setInterval(async () => {
if (sockets[index] && sockets[index].readyState === WebSocket.OPEN) {
const proxy = proxies[index % proxies.length];
const agent = useProxy ? new HttpsProxyAgent(http://${proxy.username}:${proxy.password}@${proxy.host}:${proxy.port}) : null;
  sockets[index].send(JSON.stringify({ type: "PING" }), { agent });
  logAllAccounts();
}

}, 10000);
}
function stopPinging(index) {
if ( pingIntervals[index]) {
clearInterval(pingIntervals[index]);
pingIntervals[index] = null;
}
}
process.on('SIGINT', () => {
console.log('Stopping...');
for (let i = 0; i < accounts.length; i++) {
stopPinging(i);
disconnectWebSocket(i);
}
process.exit(0);
});
function startCountdownAndPoints(index) {
clearInterval(countdownIntervals[index]);
updateCountdownAndPoints(index);
countdownIntervals[index] = setInterval(() => updateCountdownAndPoints(index), 1000);
}
async function updateCountdownAndPoints(index) {
const restartThreshold = 60000;
const now = new Date();
if (!lastUpdateds[index]) {
lastUpdateds[index] = {};
}
if (countdowns[index] === "Calculating...") {
const lastCalculatingTime = lastUpdateds[index].calculatingTime || now;
const calculatingDuration = now.getTime() - lastCalculatingTime.getTime();
if (calculatingDuration > restartThreshold) {

  restartAccountProcess(index);
  return;
}

}
if (lastUpdateds[index]) {
const nextHeartbeat = new Date(lastUpdateds[index]);
nextHeartbeat.setMinutes(nextHeartbeat.getMinutes() + 15);
const diff = nextHeartbeat.getTime() - now.getTime();
if (diff > 0) {
  const minutes = Math.floor(diff / 60000);
  const seconds = Math.floor((diff % 60000) / 1000);
  countdowns[index] = `${minutes}m ${seconds}s`;

  const maxPoints = 25;
  const timeElapsed = now.getTime() - new Date(lastUpdateds[index]).getTime();
  const timeElapsedMinutes = timeElapsed / (60 * 1000);
  let newPoints = Math.min(maxPoints, (timeElapsedMinutes / 15) * maxPoints);
  newPoints = parseFloat(newPoints.toFixed(2));

  if (Math.random() < 0.1) {
    const bonus = Math.random() * 2;
    newPoints = Math.min(maxPoints, newPoints + bonus);
    newPoints = parseFloat(newPoints.toFixed(2));
  }

  potentialPoints[index] = newPoints;
} else {
  countdowns[index] = "Calculating, it might take a minute before starting...";
  potentialPoints[index] = 25;

  lastUpdateds[index].calculatingTime = now;
}

} else {
countdowns[index] = "Calculating, it might take a minute before starting...";
potentialPoints[index] = 0;
lastUpdateds[index].calculatingTime = now;

}
logAllAccounts();
}
function restartAccountProcess(index) {
disconnectWebSocket(index);
connectWebSocket(index);
console.log(WebSocket restarted for index: ${index});
}
async function getUserId(index) {
const loginUrl = "https://ikknngrgxuxgjhplbpey.supabase.co/auth/v1/token?grant_type=password";
const proxy = proxies[index % proxies.length];
const agent = useProxy ? new HttpsProxyAgent(http://${proxy.username}:${proxy.password}@${proxy.host}:${proxy.port}) : null;
try {
const response = await axios.post(loginUrl, {
email: accounts[index].email,
password: accounts[index].password
}, {
headers: {
'Authorization': authorization,
'apikey': apikey
},
httpsAgent: agent
});
userIds[index] = response.data.user.id;
logAllAccounts();

const profileUrl = `https://ikknngrgxuxgjhplbpey.supabase.co/rest/v1/profiles?select=personal_code&id=eq.${userIds[index]}`;
const profileResponse = await axios.get(profileUrl, {
  headers: {
    'Authorization': authorization,
    'apikey': apikey
  },
  httpsAgent: agent
});

console.log(`Profile Data for Account ${index + 1}:`, profileResponse.data);
startCountdownAndPoints(index);
await connectWebSocket(index);

} catch (error) {
console.error(Error for Account ${index + 1}:, error.response ? error.response.data : error.message);
}
}
displayHeader();
for (let i = 0; i < accounts.length; i++) {
potentialPoints[i] = 0;
countdowns[i] = "Calculating...";
pointsTotals[i] = 0;
pointsToday[i] = 0;
lastUpdateds[i] = null;
messages[i] = '';
userIds[i] = null;
getUserId(i);
}
it seems to be fine


Basic Rails Container post

@rdormer hey, can you update this PR when you have some time? It'd be nice to publish it this week 🤙


Hide pods column in applications table on stage page if stage.spec.clusterName is not in-cluster


[FIX] copy of dseg.tsv following templateflow fsaverage update

see https://github.com/nipreps/smriprep/pull/393#issuecomment-2010268610

What a mess. I think I know the issue. Let me push to your branch...


setVolume() now also applies volume to audio tags

Updated setVolume() to support both video- and audio-tags.
New use-case: bandcamp.com

Hi @ThomasDecavele
I'm sorry, I somehow missed your pull request. I'll check your changes and update google webstore/firefox addons after i've tested it.
Thanks your very much for your contribution.

Hi @puddingspudding :)
It's no big deal, I was just fiddling with your plugin to change volume of audio tags as well. And thought to send it as pullrequest, but to be honest I found a solution without any duplicate code, thus way cleaner.
Just change this line:
var videoTags = document.getElementsByTagName("video");
to var videoTags = document.querySelectorAll("video, audio");
Changing that selector should be enough, right?

I don't mind duplicate code for now.


Critique on accuracy

[Enter feedback here]
In section: "Create and call a new extension method", the code boxes are marked as "C#" code, when this is "X++".
Also, there is a ttsbegin/ttscommit pair outside of the try/catch block which renders the catch useless in this example.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 541407d5-1fb2-f9ea-d7d9-1e0ab78c2793
Version Independent ID: a41c077f-a2f6-d197-f705-1809a5a5b0eb
Content: Extend Commerce Data Exchange - Real-time Service - Commerce | Dynamics 365
Content Source: articles/commerce/dev-itpro/extend-commerce-data-exchange.md
Service: **dynamics-365-commerce **
Product: ****
Technology: ****
GitHub Login: @mugunthanm
Microsoft Alias: mumani

Bug 579367 created to track this issue.

Thanks for the feedback, updated the document.


chore: update metaphysics graphql schema

Greetings human :robot: this PR was automatically created as part of metaphysics' deploy process. #nochangelog

Warnings

:warning:

✅ No changelog changes

  Generated by :no_entry_sign: dangerJS against 89316bd2f23f778ebe74d35c04421c396a8f3573


Unexpected message: No supported devices connected

So I installed PenNote as follows on my Ubuntu 20.10 system:
``bash
sudo snap install flutter --classic
cd ~/Applications
git clone https://github.com/MeshaMakes/PenNote.git
cd PenNote/pennote
flutter pub get
flutter run  # Downloads flutter first time
But there is only a message in response to the last command after which I get my shell prompt back:

```bash
No supported devices connected.

Shouldn't a widget pop up and let me edit a Markdown file?

Ping.
I'm looking forward to the next iteration of this project.


[libmicrohttpd] Add platform.h to restricted header list.

Fixes #12450.
No other ports currently publish platform.h based on the CI file list, so I did not globally invalidate cached binaries.

Related: #12434 #12084 #11208 #11208 #12342 etc.

qt5-imageformats regression related to https://github.com/microsoft/vcpkg/pull/11208#issuecomment-658575646


fix: upgrade lighthouse

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To complete the pull request process, please assign jstrachan
You can assign the PR to them by writing /assign @jstrachan in a comment when ready.
The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

/approve

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: jstrachan
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [jstrachan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: jstrachan
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [jstrachan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@jstrachan: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

gsm
251cf3d759b30a8254e3347284d25447cfe93ba4
link
/test gsm

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the jenkins-x/lighthouse repository. I understand the commands that are listed here.

@jstrachan: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

gsm
251cf3d759b30a8254e3347284d25447cfe93ba4
link
/test gsm

vault
b383b24ccb5733865b0490eb734697801585e829
link
/test vault

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the jenkins-x/lighthouse repository. I understand the commands that are listed here.


Targeting S+ (version 31 and above) requires that one of FLAG_IMMUTABLE or FLAG_MUTABLE be specified when creating a PendingIntent

After upgrading to the latest Ionic, latest Capacitor, latest Angular etc on an old app, I eventually got to the point where I was testing the background upload tasks and this error started to arise.
It should be noted I had to include:
implementation 'com.squareup.okhttp3:okhttp:4.10.0'

in my android/app/build.gradle to overcome okhttp issues with this plugin, and then the PendingIntent errors started popping up. The tasks get backgrounded, but never start uploading for real:
D/FileTransferBG: upload with id 852d3a15-d5d8-da8f-ecaf-9442cc42acbe is already exists in upload queue. ignoring re-upload request

Thanks any help is appreciated.

This is the output every time I try to upload, I'm using Ionic 6 in Angular. Everything works fine on iOS but not on Android :(
I have changed URLs and naming to protect the privacy of the project, but the rest of the output is exact.
It appears to schedule the background task but not fire it, it just gets enqueued. Pleeeease help.
2022-11-13 17:20:29.354 20671-20671/au.com.optix I/Capacitor/Console: File: http://<IP_ADDRESS>:8100/287.js - Line 1 - Msg: SENDING fileUri: /storage/emulated/0/Movies/Optix/ZpidoNGwGf.mp4
2022-11-13 17:20:29.355 20671-20671/au.com.optix V/Capacitor/Plugin: To native (Capacitor plugin): callbackId: 19470771, pluginId: Preferences, methodName: get
2022-11-13 17:20:29.355 20671-20671/au.com.optix V/Capacitor: callback: 19470771, pluginId: Preferences, methodName: get, methodData: {"key":"optix_user_token"}
2022-11-13 17:20:29.359 20671-20671/au.com.optix V/Capacitor/Plugin: To native (Capacitor plugin): callbackId: 19470772, pluginId: Filesystem, methodName: readFile
2022-11-13 17:20:29.359 20671-20671/au.com.optix V/Capacitor: callback: 19470772, pluginId: Filesystem, methodName: readFile, methodData: {"path":"file:\/\/\/storage\/emulated\/0\/Movies\/Optix\/ZpidoNGwGf.mp4"}
2022-11-13 17:20:29.405 20671-20671/au.com.optix V/Capacitor/Plugin: To native (Cordova plugin): callbackId: FileTransferBackground982115603, service: FileTransferBackground, action: initManager, actionArgs: []
2022-11-13 17:20:29.445 20671-20671/au.com.optix D/FileTransferBG: created FileTransfer working directory
2022-11-13 17:20:29.448 20671-20671/au.com.optix I/Capacitor/Console: File: http://<IP_ADDRESS>:8100/287.js - Line 1 - Msg: STARTING BACKGROUND UPLOAD
2022-11-13 17:20:29.449 20671-20671/au.com.optix V/Capacitor/Plugin: To native (Cordova plugin): callbackId: FileTransferBackground982115604, service: FileTransferBackground, action: startUpload, actionArgs: [{"id":"optix_vid_a656b034-9553-319b-2b00-b4c5219c7e26","filePath":"\/storage\/emulated\/0\/Movies\/Optix\/ZpidoNGwGf.mp4","fileKey":"video_file","serverUrl":"https:\/\/portal.privateurl.com.au\/auth\/api\/videos\/upload","notificationTitle":"Optix Uploads","headers":{"Authorization":"Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJhdWQiOiIyIiwianRpIjoiNjA0NzQ5Nzg2YWRlMzlmZmViMWQ1YTMxZTkyZDcyMzA4NzNlYjU4ZDM0MWFiNTkxOGYxYTBjMTQxY2UwOWU5YWQ4YjdjYWJmNjE3ZGUxZWYiLCJpYXQiOjE2NjgzMjg0NjMuNTM3NDgyLCJuYmYiOjE2NjgzMjg0NjMuNTM3NDg3LCJleHAiOjE2ODM5NjY4NjMuNTMwMjY5LCJzdWIiOiI5Iiwic2NvcGVzIjpbXX0.KDmTTvswCeHIw5YBUzNZtnekAQTeikURLNqVgEQW-zAnBrR6IvIwOmPTcRNOGLGrKHBlCVeHCqH264qpFjifCgAYliyOoGAAaexecQjOS8B3yuKhoHmRaZb73zqzVz5b_hoyqWVig15ozgjaYOA1KVTM-T57DCAE_JbRH0Z6Ch4key4K8DWBdE3NCWySR89v2KesVM4ofop-LfWmGNK_MW3wTOzrqXvW_vdSKHX1TsJ9NXS65umgF9mEvS6WmKZDU5yni0r1Kp491cpJtF3QBlBIb4d3N0w_akCLVxlmEmZO3U_fnswJ1PM9tGFAwyd6AcYoapYdvCKWCDiOshWDg7905JSde2hD2Y_mml4deFWJ8EiKkfO69cJrYsjBoT5krC6uUzVxMrETf4ArrL0EG0c3YITBxl5g8XBgmxsdL8gVV-FDVVOugmKDMoR8oolFTOEYju9DRkoETtvJl6nPwASnTYO04zglbtG1YuuSUMjXMh44UtxgiUQ5qZ9EvCy8SAEkvi6XwZm9FjVxu9nIrIH5LukjRB2P6ciXsny5vtqiI1MNhWXHDIy6DyP2foZF__ywLqLN-LVDHZa0qJFeMVXA2Oew57OdgcZrdhlo4bVlO560KK-8dimmJ8KPDS8_iErItHikhbIpRVRH9DxdiUpOmhPRHrENq6_CQV5bJbg"},"parameters":{"customer_id":"101","customer_url":"ZpidoNGwGf","filename":"ZpidoNGwGf.mp4"},"showNotification":true}]
2022-11-13 17:20:29.450 20671-20671/au.com.optix D/FileTransferBG: adding upload optix_vid_a656b034-9553-319b-2b00-b4c5219c7e26
2022-11-13 17:20:29.453 20671-20671/au.com.optix D/CompatibilityChangeReporter: Compat change id reported: 160794467; UID 10045; state: ENABLED
2022-11-13 17:20:29.453 20671-20671/au.com.optix I/Capacitor/Console: File: http://<IP_ADDRESS>:8100/287.js - Line 1 - Msg: STARTED!!!! {"_handlers":{"progress":[],"success":[],"error":[]},"options":{}}
2022-11-13 17:20:29.454 20671-20671/au.com.optix D/FileTransferBG: detected network change, Connected:true
2022-11-13 17:20:29.456 20671-20671/au.com.optix D/FileTransferBG: upload with id optix_vid_a656b034-9553-319b-2b00-b4c5219c7e26 is already exists in upload queue. ignoring re-upload request
2022-11-13 17:20:33.833 20671-21127/au.com.optix D/BufferPoolAccessor2.0: bufferpool2 0xb400007bfa396e48 : 0(0 size) total buffers - 0(0 size) used buffers - 0/6 (recycle/alloc) - 6/81 (fetch/transfer)
2022-11-13 17:20:33.834 20671-21127/au.com.optix D/BufferPoolAccessor2.0: evictor expired: 1, evicted: 1

Hello @gbrits,
Can you send me your package.json, config.xml and your payload that you are sending when uploading your file.

My config.xml:
<?xml version='1.0' encoding='utf-8'?>
<widget version="1.0.0" xmlns="http://www.w3.org/ns/widgets" xmlns:cdv="http://cordova.apache.org/ns/1.0">
  <access origin="*" />
  <feature name="FileTransferBackground">
    <param name="ios-package" value="FileTransferBackground"/>
  </feature>
  <feature name="Camera">
    <param name="ios-package" value="CDVCamera"/>
  </feature>
  <feature name="File">
    <param name="ios-package" value="CDVFile"/>
    <param name="onload" value="true"/>
  </feature>
  <feature name="CDVOrientation">
    <param name="ios-package" value="CDVOrientation"/>
  </feature>
  <feature name="VideoEditor">
    <param name="ios-package" value="VideoEditor"/>
  </feature>
  <feature name="VideoCapturePlus">
    <param name="ios-package" value="VideoCapturePlus"/>
  </feature>
</widget>

My package.json:
{
  "name": "HiddenName",
  "version": "0.0.5",
  "author": "Hidden Author",
  "homepage": "https://hidden.author/",
  "scripts": {
    "ng": "ng",
    "start": "ng serve",
    "build": "ng build",
    "test": "ng test",
    "lint": "ng lint",
    "e2e": "ng e2e"
  },
  "private": true,
  "dependencies": {
    "@angular/common": "^14.2.10",
    "@angular/core": "^14.2.10",
    "@angular/forms": "^14.2.10",
    "@angular/platform-browser": "^14.2.10",
    "@angular/platform-browser-dynamic": "^14.2.10",
    "@angular/router": "^14.2.10",
    "@awesome-cordova-plugins/background-upload": "^6.2.0",
    "@awesome-cordova-plugins/core": "^6.2.0",
    "@awesome-cordova-plugins/file": "^6.2.0",
    "@awesome-cordova-plugins/video-editor": "^6.2.0",
    "@capacitor-community/camera-preview": "4.0.0",
    "@capacitor/android": "4.4.0",
    "@capacitor/core": "^4.4.0",
    "@capacitor/dialog": "^4.0.1",
    "@capacitor/filesystem": "^4.1.3",
    "@capacitor/ios": "^4.4.0",
    "@capacitor/preferences": "^4.0.1",
    "@capacitor/status-bar": "^4.0.1",
    "@ionic-native/core": "^5.36.0",
    "@ionic-native/file-transfer": "^5.36.0",
    "@ionic-native/ionic-webview": "^5.36.0",
    "@ionic-native/screen-orientation": "^5.36.0",
    "@ionic-native/splash-screen": "^5.36.0",
    "@ionic-native/video-capture-plus": "^5.36.0",
    "@ionic-native/video-editor": "^5.36.0",
    "@ionic/angular": "^6.3.6",
    "chart.js": "^2.9.4",
    "cordova-plugin-androidx": "^3.0.0",
    "cordova-plugin-androidx-adapter": "^1.1.3",
    "cordova-plugin-background-upload": "^1.1.0",
    "cordova-plugin-camera": "^6.0.0",
    "cordova-plugin-compat": "^1.2.0",
    "cordova-plugin-file": "^7.0.0",
    "cordova-plugin-file-transfer": "^1.7.1",
    "cordova-plugin-ionic-webview": "^5.0.0",
    "cordova-plugin-screen-orientation": "^3.0.2",
    "cordova-plugin-video-editor": "^1.1.3",
    "date-fns": "^2.28.0",
    "es6-promise-plugin": "^4.2.2",
    "guid-typescript": "^1.0.9",
    "jszip": "^3.10.0",
    "mxms-vid-capture": "^0.0.4",
    "nanoid": "^3.3.4",
    "ng2-search-filter": "^0.5.1",
    "root-require": "^0.3.1",
    "rxjs": "^6.5.5",
    "tslib": "^2.1.0",
    "tus-js-client": "^2.2.0",
    "zone.js": "~0.12.0"
  },
  "devDependencies": {
    "@angular-devkit/build-angular": "^14.2.9",
    "@angular/cli": "^10.0.5",
    "@angular/compiler": "^14.2.10",
    "@angular/compiler-cli": "^14.2.10",
    "@angular/language-service": "^14.2.10",
    "@capacitor/cli": "^4.4.0",
    "@ionic/angular-toolkit": "^2.3.0",
    "@types/jasmine": "^4.0.3",
    "@types/jasminewd2": "~2.0.10",
    "@types/node": "^17.0.36",
    "codelyzer": "^6.0.2",
    "jasmine-core": "~4.1.1",
    "jasmine-spec-reporter": "~7.0.0",
    "jetifier": "^2.0.0",
    "karma": "~6.3.20",
    "karma-chrome-launcher": "~3.1.1",
    "karma-coverage-istanbul-reporter": "~3.0.3",
    "karma-jasmine": "~5.0.1",
    "karma-jasmine-html-reporter": "^2.0.0",
    "protractor": "~7.0.0",
    "ts-node": "~8.3.0",
    "tslint": "~6.1.0",
    "typescript": "^4.8.4"
  },
  "description": "An Ionic project"
}

The payload:
2022-11-13 17:20:29.449 20671-20671/au.com.optix V/Capacitor/Plugin: To native (Cordova plugin): callbackId: FileTransferBackground982115604, service: FileTransferBackground, action: startUpload, actionArgs: [{"id":"optix_vid_a656b034-9553-319b-2b00-b4c5219c7e26","filePath":"\/storage\/emulated\/0\/Movies\/Optix\/ZpidoNGwGf.mp4","fileKey":"video_file","serverUrl":"https:\/\/portal.privateurl.com.au\/auth\/api\/videos\/upload","notificationTitle":"Optix Uploads","headers":{"Authorization":"Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJhdWQiOiIyIiwianRpIjoiNjA0NzQ5Nzg2YWRlMzlmZmViMWQ1YTMxZTkyZDcyMzA4NzNlYjU4ZDM0MWFiNTkxOGYxYTBjMTQxY2UwOWU5YWQ4YjdjYWJmNjE3ZGUxZWYiLCJpYXQiOjE2NjgzMjg0NjMuNTM3NDgyLCJuYmYiOjE2NjgzMjg0NjMuNTM3NDg3LCJleHAiOjE2ODM5NjY4NjMuNTMwMjY5LCJzdWIiOiI5Iiwic2NvcGVzIjpbXX0.KDmTTvswCeHIw5YBUzNZtnekAQTeikURLNqVgEQW-zAnBrR6IvIwOmPTcRNOGLGrKHBlCVeHCqH264qpFjifCgAYliyOoGAAaexecQjOS8B3yuKhoHmRaZb73zqzVz5b_hoyqWVig15ozgjaYOA1KVTM-T57DCAE_JbRH0Z6Ch4key4K8DWBdE3NCWySR89v2KesVM4ofop-LfWmGNK_MW3wTOzrqXvW_vdSKHX1TsJ9NXS65umgF9mEvS6WmKZDU5yni0r1Kp491cpJtF3QBlBIb4d3N0w_akCLVxlmEmZO3U_fnswJ1PM9tGFAwyd6AcYoapYdvCKWCDiOshWDg7905JSde2hD2Y_mml4deFWJ8EiKkfO69cJrYsjBoT5krC6uUzVxMrETf4ArrL0EG0c3YITBxl5g8XBgmxsdL8gVV-FDVVOugmKDMoR8oolFTOEYju9DRkoETtvJl6nPwASnTYO04zglbtG1YuuSUMjXMh44UtxgiUQ5qZ9EvCy8SAEkvi6XwZm9FjVxu9nIrIH5LukjRB2P6ciXsny5vtqiI1MNhWXHDIy6DyP2foZF__ywLqLN-LVDHZa0qJFeMVXA2Oew57OdgcZrdhlo4bVlO560KK-8dimmJ8KPDS8_iErItHikhbIpRVRH9DxdiUpOmhPRHrENq6_CQV5bJbg"},"parameters":{"customer_id":"101","customer_url":"ZpidoNGwGf","filename":"ZpidoNGwGf.mp4"},"showNotification":true}]

Ok so since upgrading to Angular 11.0 and updating a bunch of the toolkits as a result, I'm now getting the unable to find file error when trying to upload. Never been so happy to see an error!
I also think I was using cordova-plugin-background-upload and had to delete that, to use this library.

Attempting to upload via fileUri from a transcoder. Will update here with the fix for that.

Updated package.json:
"dependencies": {
    "@angular/common": "^15.0.0",
    "@angular/core": "^15.0.0",
    "@angular/forms": "^15.0.0",
    "@angular/platform-browser": "^15.0.0",
    "@angular/platform-browser-dynamic": "^15.0.0",
    "@angular/router": "^15.0.0",
    "@awesome-cordova-plugins/background-mode": "^6.2.0",
    "@awesome-cordova-plugins/background-upload": "^6.2.0",
    "@awesome-cordova-plugins/core": "^6.2.0",
    "@awesome-cordova-plugins/file": "^6.2.0",
    "@awesome-cordova-plugins/video-editor": "^6.2.0",
    "@capacitor-community/camera-preview": "4.0.0",
    "@capacitor/android": "4.5.0",
    "@capacitor/core": "^4.4.0",
    "@capacitor/dialog": "^4.0.1",
    "@capacitor/filesystem": "^4.1.3",
    "@capacitor/ios": "^4.4.0",
    "@capacitor/preferences": "^4.0.1",
    "@capacitor/status-bar": "^4.0.1",
    "@ionic-native/core": "^5.36.0",
    "@ionic-native/file-transfer": "^5.36.0",
    "@ionic-native/ionic-webview": "^5.36.0",
    "@ionic-native/screen-orientation": "^5.36.0",
    "@ionic-native/splash-screen": "^5.36.0",
    "@ionic-native/video-capture-plus": "^5.36.0",
    "@ionic-native/video-editor": "^5.36.0",
    "@ionic/angular": "^6.3.6",
    "@spoonconsulting/cordova-plugin-background-upload": "^4.0.7",
    "chart.js": "^2.9.4",
    "cordova-plugin-androidx": "^3.0.0",
    "cordova-plugin-androidx-adapter": "^1.1.3",
    "cordova-plugin-camera": "^6.0.0",
    "cordova-plugin-compat": "^1.2.0",
    "cordova-plugin-file": "^7.0.0",
    "cordova-plugin-ionic-webview": "^5.0.0",
    "cordova-plugin-ns0m-file-transfer": "^2.0.0-7",
    "cordova-plugin-screen-orientation": "^3.0.2",
    "cordova-plugin-video-editor": "^1.1.3",
    "date-fns": "^2.28.0",
    "es6-promise-plugin": "^4.2.2",
    "guid-typescript": "^1.0.9",
    "jszip": "^3.10.0",
    "mxms-vid-capture": "^0.0.4",
    "nanoid": "^3.3.4",
    "ng2-search-filter": "^0.5.1",
    "root-require": "^0.3.1",
    "rxjs": "^6.5.5",
    "tslib": "^2.0.0",
    "tus-js-client": "^2.2.0",
    "zone.js": "~0.12.0"
  },
  "devDependencies": {
    "@angular-devkit/build-angular": "^15.0.0",
    "@angular/cli": "^11.2.19",
    "@angular/compiler": "^15.0.0",
    "@angular/compiler-cli": "^15.0.0",
    "@angular/language-service": "^15.0.0",
    "@capacitor/cli": "^4.4.0",
    "@ionic/angular-toolkit": "^7.0.0",
    "@types/jasmine": "~3.6.0",
    "@types/jasminewd2": "~2.0.10",
    "@types/node": "^17.0.36",
    "codelyzer": "^6.0.0",
    "jasmine-core": "^4.5.0",
    "jasmine-spec-reporter": "~5.0.0",
    "jetifier": "^2.0.0",
    "karma": "~6.3.20",
    "karma-chrome-launcher": "~3.1.0",
    "karma-coverage-istanbul-reporter": "~3.0.3",
    "karma-jasmine": "^5.1.0",
    "karma-jasmine-html-reporter": "^1.5.0",
    "protractor": "~7.0.0",
    "ts-node": "~8.3.0",
    "tslint": "~6.1.0",
    "typescript": "^4.8.4"
  },

The URI that is failing to upload:
FileURI:
/storage/emulated/0/Android/data/au.com.private-url/files/files/videos/-O6RLl1L2S.mp4

Hello @gbrits,
I am seeing in your package.json that you are using not the latest version: 4.0.7
Make sure to include the installation like this in you package.json:
"@spoonconsulting/cordova-plugin-background-upload": "4.0.7"


use dimensions instead of float for pathWidth

Just corrected pathWidth to accept dimensions instead of float.

Thanks,
This should be merged today.

Thanks!

Available from 1.0.7


Missing files due to improper folder specification

Hi,
I am trying to run the example provided for mgatk bcall under mgatk/tests, which results in an error due to missing files. Within bc1d/final/, there is only chrM_refAllele.txt generated. I get the same error when trying to run the genotyping for my own 10x-scATAC data. Do you have an idea what might be the problem here?
cd mgatk/tests
mgatk bcall -i barcode/test_barcode.bam -n bc1 -o bc1d -bt CB -b barcode/test_barcodes.txt -z
Fri Jun 12 10:44:30 CEST 2020: mgatk v0.5.3
Fri Jun 12 10:44:30 CEST 2020: Found bam file: barcode/test_barcode.bam for genotyping.
Fri Jun 12 10:44:30 CEST 2020: Found file of barcodes to be parsed: barcode/test_barcodes.txt
Fri Jun 12 10:44:30 CEST 2020: User specified mitochondrial genome matches .bam file
Fri Jun 12 10:44:33 CEST 2020: Finished determining/splitting barcodes for genotyping.
Fri Jun 12 10:44:33 CEST 2020: Genotyping samples with 1 threads
Error in checkGrep(grep(".A.txt", files)) :
Improper folder specification; file missing / extra file present. See documentation
Calls: importMito -> checkGrep
Execution halted
Thanks and best wishes,
Malte

Could you please sense the version of the software used and the intermediate output files contained in the folder?

On Jun 12, 2020, at 4:58 AM, Malte Simon<EMAIL_ADDRESS>wrote:

Hi,
I am trying to run the example provided for mgatk bcall under mgatk/tests, which results in an error due to missing files. Within bc1d/final/, there is only chrM_refAllele.txt generated. I get the same error when trying to run the genotyping for my own 10x-scATAC data. Do you have an idea what might be the problem here?
cd mgatk/tests
mgatk bcall -i barcode/test_barcode.bam -n bc1 -o bc1d -bt CB -b barcode/test_barcodes.txt -z
Fri Jun 12 10:44:30 CEST 2020: mgatk v0.5.3
Fri Jun 12 10:44:30 CEST 2020: Found bam file: barcode/test_barcode.bam for genotyping.
Fri Jun 12 10:44:30 CEST 2020: Found file of barcodes to be parsed: barcode/test_barcodes.txt
Fri Jun 12 10:44:30 CEST 2020: User specified mitochondrial genome matches .bam file
Fri Jun 12 10:44:33 CEST 2020: Finished determining/splitting barcodes for genotyping.
Fri Jun 12 10:44:33 CEST 2020: Genotyping samples with 1 threads
Error in checkGrep(grep(".A.txt", files)) :
Improper folder specification; file missing / extra file present. See documentation
Calls: importMito -> checkGrep
Execution halted
Thanks and best wishes,
Malte
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

@mathosi thanks for the email The eror has to do with an incompatibility issue in snakemake that was recently addressed in version 0.5.5 that has now been pushed to PyPi. Please update and rerun, and I would anticipate that the issue should be addressed.

Thank you, updating to version 0.5.5 solved the problem!

Hi,
I'm getting the same error under version 0.5.6 (snakemake version 5.20.1):
$ mgatk bcall -i test/test_barcode.bam -n bc1 -o bc1d -bt CB -b test/test_barcodes.txt -z
Wed Aug 05 14:50:49 CEST 2020: mgatk v0.5.6
Wed Aug 05 14:50:50 CEST 2020: Found bam file: test/test_barcode.bam for genotyping.
Wed Aug 05 14:50:50 CEST 2020: Found file of barcodes to be parsed: test/test_barcodes.txt
Wed Aug 05 14:50:50 CEST 2020: User specified mitochondrial genome matches .bam file
Wed Aug 05 14:50:53 CEST 2020: Finished determining/splitting barcodes for genotyping.
Wed Aug 05 14:50:53 CEST 2020: Genotyping samples with 2 threads
Error in checkGrep(grep(".A.txt", files)) :
  Improper folder specification; file missing / extra file present. See documentation
Calls: importMito -> checkGrep
Execution halted

any ideas? thanks!
Benedikt

Can you post the contents of the folder ? (ls -lR)

On Aug 5, 2020, at 5:52 AM, Benedikt Obermayer<EMAIL_ADDRESS>wrote:

Hi,
I'm getting the same error under version 0.5.6 (snakemake version 5.20.1):
$ mgatk bcall -i test/test_barcode.bam -n bc1 -o bc1d -bt CB -b test/test_barcodes.txt -z
Wed Aug 05 14:50:49 CEST 2020: mgatk v0.5.6
Wed Aug 05 14:50:50 CEST 2020: Found bam file: test/test_barcode.bam for genotyping.
Wed Aug 05 14:50:50 CEST 2020: Found file of barcodes to be parsed: test/test_barcodes.txt
Wed Aug 05 14:50:50 CEST 2020: User specified mitochondrial genome matches .bam file
Wed Aug 05 14:50:53 CEST 2020: Finished determining/splitting barcodes for genotyping.
Wed Aug 05 14:50:53 CEST 2020: Genotyping samples with 2 threads
Error in checkGrep(grep(".A.txt", files)) :
Improper folder specification; file missing / extra file present. See documentation
Calls: importMito -> checkGrep
Execution halted
any ideas? thanks!
Benedikt
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.

hi, sure:
$ ls -lR test
test:
total 7169
-rw-r--r-- 1 obermayb_c hpc-ag-cubi 7333460 Aug  5 14:49 test_barcode.bam
-rw-r--r-- 1 obermayb_c hpc-ag-cubi     864 Aug  5 14:49 test_barcode.bam.bai
-rw-r--r-- 1 obermayb_c hpc-ag-cubi      57 Aug  5 14:49 test_barcodes.txt

and
$ ls -lR bc1d
bc1d:
total 3
drwxr-sr-x 2 obermayb_c hpc-ag-cubi 4096 Aug  6 08:42 fasta
drwxr-sr-x 2 obermayb_c hpc-ag-cubi 4096 Aug  6 08:42 final
drwxr-sr-x 4 obermayb_c hpc-ag-cubi 4096 Aug  6 08:42 logs
drwxr-sr-x 3 obermayb_c hpc-ag-cubi 4096 Aug  6 08:42 qc
drwxr-sr-x 5 obermayb_c hpc-ag-cubi 4096 Aug  6 08:42 temp

bc1d/fasta:
total 257
-rw-r--r-- 1 obermayb_c hpc-ag-cubi 16907 Aug  6 08:42 chrM.fasta
-rw-r--r-- 1 obermayb_c hpc-ag-cubi    19 Aug  6 08:42 chrM.fasta.fai

bc1d/final:
total 256
-rw-r--r-- 1 obermayb_c hpc-ag-cubi 121446 Aug  6 08:42 chrM_refAllele.txt

bc1d/logs:
total 259
-rw-r--r-- 1 obermayb_c hpc-ag-cubi  438 Aug  6 08:42 base.mgatk.log
-rw-r--r-- 1 obermayb_c hpc-ag-cubi  452 Aug  6 08:42 bc1.parameters.txt
-rw-r--r-- 1 obermayb_c hpc-ag-cubi 2002 Aug  6 08:42 bc1.snakemake_gather.log
-rw-r--r-- 1 obermayb_c hpc-ag-cubi 5171 Aug  6 08:42 bc1.snakemake_scatter.log
drwxr-sr-x 2 obermayb_c hpc-ag-cubi 4096 Aug  6 08:42 filterlogs
drwxr-sr-x 2 obermayb_c hpc-ag-cubi 4096 Aug  6 08:42 rmdupslogs

bc1d/logs/filterlogs:
total 1
-rw-r--r-- 1 obermayb_c hpc-ag-cubi 21 Aug  6 08:42 CTAACTTAGAGCCACA-1.filter.log
-rw-r--r-- 1 obermayb_c hpc-ag-cubi 21 Aug  6 08:42 GCCTAGGCAGTTCGGC-1.filter.log

bc1d/logs/rmdupslogs:
total 0

bc1d/qc:
total 1
drwxr-sr-x 2 obermayb_c hpc-ag-cubi 4096 Aug  6 08:42 quality

bc1d/qc/quality:
total 0

bc1d/temp:
total 2
drwxr-sr-x 2 obermayb_c hpc-ag-cubi 4096 Aug  6 08:42 barcoded_bams
drwxr-sr-x 2 obermayb_c hpc-ag-cubi 4096 Aug  6 08:42 quality
drwxr-sr-x 2 obermayb_c hpc-ag-cubi 4096 Aug  6 08:42 temp_bam

bc1d/temp/barcoded_bams:
total 7682
-rw-r--r-- 1 obermayb_c hpc-ag-cubi 2765493 Aug  6 08:42 CACCACTAGGAGGCGA-1.bam
-rw-r--r-- 1 obermayb_c hpc-ag-cubi     808 Aug  6 08:42 CACCACTAGGAGGCGA-1.bam.bai
-rw-r--r-- 1 obermayb_c hpc-ag-cubi 2429008 Aug  6 08:42 CTAACTTAGAGCCACA-1.bam
-rw-r--r-- 1 obermayb_c hpc-ag-cubi     808 Aug  6 08:42 CTAACTTAGAGCCACA-1.bam.bai
-rw-r--r-- 1 obermayb_c hpc-ag-cubi 2116311 Aug  6 08:42 GCCTAGGCAGTTCGGC-1.bam
-rw-r--r-- 1 obermayb_c hpc-ag-cubi     792 Aug  6 08:42 GCCTAGGCAGTTCGGC-1.bam.bai

bc1d/temp/quality:
total 0

bc1d/temp/temp_bam:
total 9729
-rw-r--r-- 1 obermayb_c hpc-ag-cubi 2429008 Aug  6 08:42 CTAACTTAGAGCCACA-1.temp0.bam
-rw-r--r-- 1 obermayb_c hpc-ag-cubi 2429112 Aug  6 08:42 CTAACTTAGAGCCACA-1.temp1.bam
-rw-r--r-- 1 obermayb_c hpc-ag-cubi     808 Aug  6 08:42 CTAACTTAGAGCCACA-1.temp1.bam.bai
-rw-r--r-- 1 obermayb_c hpc-ag-cubi 2116311 Aug  6 08:42 GCCTAGGCAGTTCGGC-1.temp0.bam
-rw-r--r-- 1 obermayb_c hpc-ag-cubi 2116416 Aug  6 08:42 GCCTAGGCAGTTCGGC-1.temp1.bam
-rw-r--r-- 1 obermayb_c hpc-ag-cubi     792 Aug  6 08:42 GCCTAGGCAGTTCGGC-1.temp1.bam.bai

From this directory, can you
cat bc1d/logs/*.log

There should be some level of error message there

On Aug 5, 2020, at 11:44 PM, Benedikt Obermayer<EMAIL_ADDRESS>wrote:
hi, sure:
$ ls -lR test
test:
total 7169
-rw-r--r-- 1 obermayb_c hpc-ag-cubi 7333460 Aug  5 14:49 test_barcode.bam
-rw-r--r-- 1 obermayb_c hpc-ag-cubi     864 Aug  5 14:49 test_barcode.bam.bai
-rw-r--r-- 1 obermayb_c hpc-ag-cubi      57 Aug  5 14:49 test_barcodes.txt
and
$ ls -lR bc1d
bc1d:
total 3
drwxr-sr-x 2 obermayb_c hpc-ag-cubi 4096 Aug  6 08:42 fasta
drwxr-sr-x 2 obermayb_c hpc-ag-cubi 4096 Aug  6 08:42 final
drwxr-sr-x 4 obermayb_c hpc-ag-cubi 4096 Aug  6 08:42 logs
drwxr-sr-x 3 obermayb_c hpc-ag-cubi 4096 Aug  6 08:42 qc
drwxr-sr-x 5 obermayb_c hpc-ag-cubi 4096 Aug  6 08:42 temp
bc1d/fasta:
total 257
-rw-r--r-- 1 obermayb_c hpc-ag-cubi 16907 Aug  6 08:42 chrM.fasta
-rw-r--r-- 1 obermayb_c hpc-ag-cubi    19 Aug  6 08:42 chrM.fasta.fai
bc1d/final:
total 256
-rw-r--r-- 1 obermayb_c hpc-ag-cubi 121446 Aug  6 08:42 chrM_refAllele.txt
bc1d/logs:
total 259
-rw-r--r-- 1 obermayb_c hpc-ag-cubi  438 Aug  6 08:42 base.mgatk.log
-rw-r--r-- 1 obermayb_c hpc-ag-cubi  452 Aug  6 08:42 bc1.parameters.txt
-rw-r--r-- 1 obermayb_c hpc-ag-cubi 2002 Aug  6 08:42 bc1.snakemake_gather.log
-rw-r--r-- 1 obermayb_c hpc-ag-cubi 5171 Aug  6 08:42 bc1.snakemake_scatter.log
drwxr-sr-x 2 obermayb_c hpc-ag-cubi 4096 Aug  6 08:42 filterlogs
drwxr-sr-x 2 obermayb_c hpc-ag-cubi 4096 Aug  6 08:42 rmdupslogs
bc1d/logs/filterlogs:
total 1
-rw-r--r-- 1 obermayb_c hpc-ag-cubi 21 Aug  6 08:42 CTAACTTAGAGCCACA-1.filter.log
-rw-r--r-- 1 obermayb_c hpc-ag-cubi 21 Aug  6 08:42 GCCTAGGCAGTTCGGC-1.filter.log
bc1d/logs/rmdupslogs:
total 0
bc1d/qc:
total 1
drwxr-sr-x 2 obermayb_c hpc-ag-cubi 4096 Aug  6 08:42 quality
bc1d/qc/quality:
total 0
bc1d/temp:
total 2
drwxr-sr-x 2 obermayb_c hpc-ag-cubi 4096 Aug  6 08:42 barcoded_bams
drwxr-sr-x 2 obermayb_c hpc-ag-cubi 4096 Aug  6 08:42 quality
drwxr-sr-x 2 obermayb_c hpc-ag-cubi 4096 Aug  6 08:42 temp_bam
bc1d/temp/barcoded_bams:
total 7682
-rw-r--r-- 1 obermayb_c hpc-ag-cubi 2765493 Aug  6 08:42 CACCACTAGGAGGCGA-1.bam
-rw-r--r-- 1 obermayb_c hpc-ag-cubi     808 Aug  6 08:42 CACCACTAGGAGGCGA-1.bam.bai
-rw-r--r-- 1 obermayb_c hpc-ag-cubi 2429008 Aug  6 08:42 CTAACTTAGAGCCACA-1.bam
-rw-r--r-- 1 obermayb_c hpc-ag-cubi     808 Aug  6 08:42 CTAACTTAGAGCCACA-1.bam.bai
-rw-r--r-- 1 obermayb_c hpc-ag-cubi 2116311 Aug  6 08:42 GCCTAGGCAGTTCGGC-1.bam
-rw-r--r-- 1 obermayb_c hpc-ag-cubi     792 Aug  6 08:42 GCCTAGGCAGTTCGGC-1.bam.bai
bc1d/temp/quality:
total 0
bc1d/temp/temp_bam:
total 9729
-rw-r--r-- 1 obermayb_c hpc-ag-cubi 2429008 Aug  6 08:42 CTAACTTAGAGCCACA-1.temp0.bam
-rw-r--r-- 1 obermayb_c hpc-ag-cubi 2429112 Aug  6 08:42 CTAACTTAGAGCCACA-1.temp1.bam
-rw-r--r-- 1 obermayb_c hpc-ag-cubi     808 Aug  6 08:42 CTAACTTAGAGCCACA-1.temp1.bam.bai
-rw-r--r-- 1 obermayb_c hpc-ag-cubi 2116311 Aug  6 08:42 GCCTAGGCAGTTCGGC-1.temp0.bam
-rw-r--r-- 1 obermayb_c hpc-ag-cubi 2116416 Aug  6 08:42 GCCTAGGCAGTTCGGC-1.temp1.bam
-rw-r--r-- 1 obermayb_c hpc-ag-cubi     792 Aug  6 08:42 GCCTAGGCAGTTCGGC-1.temp1.bam.bai
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub https://github.com/caleblareau/mgatk/issues/21#issuecomment-669737603, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD32FYO47IMQX53SXGUHAEDR7JGMTANCNFSM4N4EM3NQ.

yes of course.
$ cat bc1d/logs/bc1.snakemake_scatter.log
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 2
Rules claiming more threads will be scaled down.
Job counts:
	count	jobs
	1	all
	1	make_sample_list
	3	process_one_sample
	5

[Thu Aug  6 08:42:34 2020]
rule process_one_sample:
    input: bc1d/.internal/samples/GCCTAGGCAGTTCGGC-1.bam.txt
    output: bc1d/temp/ready_bam/GCCTAGGCAGTTCGGC-1.qc.bam, bc1d/temp/ready_bam/GCCTAGGCAGTTCGGC-1.qc.bam.bai, bc1d/qc/depth/GCCTAGGCAGTTCGGC-1.depth.txt, bc1d/temp/sparse_matrices/GCCTAGGCAGTTCGGC-1.A.txt, bc1d/temp/sparse_matrices/GCCTAGGCAGTTCGGC-1.C.txt, bc1d/temp/sparse_matrices/GCCTAGGCAGTTCGGC-1.G.txt, bc1d/temp/sparse_matrices/GCCTAGGCAGTTCGGC-1.T.txt, bc1d/temp/sparse_matrices/GCCTAGGCAGTTCGGC-1.coverage.txt
    jobid: 2
    wildcards: sample=GCCTAGGCAGTTCGGC-1

[Thu Aug  6 08:42:34 2020]
rule process_one_sample:
    input: bc1d/.internal/samples/CTAACTTAGAGCCACA-1.bam.txt
    output: bc1d/temp/ready_bam/CTAACTTAGAGCCACA-1.qc.bam, bc1d/temp/ready_bam/CTAACTTAGAGCCACA-1.qc.bam.bai, bc1d/qc/depth/CTAACTTAGAGCCACA-1.depth.txt, bc1d/temp/sparse_matrices/CTAACTTAGAGCCACA-1.A.txt, bc1d/temp/sparse_matrices/CTAACTTAGAGCCACA-1.C.txt, bc1d/temp/sparse_matrices/CTAACTTAGAGCCACA-1.G.txt, bc1d/temp/sparse_matrices/CTAACTTAGAGCCACA-1.T.txt, bc1d/temp/sparse_matrices/CTAACTTAGAGCCACA-1.coverage.txt
    jobid: 3
    wildcards: sample=CTAACTTAGAGCCACA-1

Job counts:
	count	jobs
	1	process_one_sample
	1
Job counts:
	count	jobs
	1	process_one_sample
	1
Traceback (most recent call last):
  File "/fast/work/users/obermayb_c/scratch/miniconda/envs/mgatk/lib/python3.6/site-packages/mgatk/bin/python/oneSample.py", line 80, in <module>
    pysam.index(outputbam)
  File "/fast/users/obermayb_c/scratch/miniconda/envs/mgatk/lib/python3.6/site-packages/pysam/utils.py", line 61, in __call__
    save_stdout=kwargs.get("save_stdout", None))
  File "pysam/libcutils.pyx", line 293, in pysam.libcutils._pysam_dispatch
OSError: No such file or directory: 'bc1d/temp/ready_bam/GCCTAGGCAGTTCGGC-1.qc.bam'
Traceback (most recent call last):
  File "/fast/work/users/obermayb_c/scratch/miniconda/envs/mgatk/lib/python3.6/site-packages/mgatk/bin/python/oneSample.py", line 80, in <module>
    pysam.index(outputbam)
  File "/fast/users/obermayb_c/scratch/miniconda/envs/mgatk/lib/python3.6/site-packages/pysam/utils.py", line 61, in __call__
    save_stdout=kwargs.get("save_stdout", None))
  File "pysam/libcutils.pyx", line 293, in pysam.libcutils._pysam_dispatch
OSError: No such file or directory: 'bc1d/temp/ready_bam/CTAACTTAGAGCCACA-1.qc.bam'
MissingOutputException in line 21 of /fast/work/users/obermayb_c/scratch/miniconda/envs/mgatk/lib/python3.6/site-packages/mgatk/bin/snake/Snakefile.Scatter:
Job completed successfully, but some output files are missing. Missing files after 5 seconds:
bc1d/temp/ready_bam/CTAACTTAGAGCCACA-1.qc.bam
bc1d/temp/ready_bam/CTAACTTAGAGCCACA-1.qc.bam.bai
bc1d/qc/depth/CTAACTTAGAGCCACA-1.depth.txt
bc1d/temp/sparse_matrices/CTAACTTAGAGCCACA-1.A.txt
bc1d/temp/sparse_matrices/CTAACTTAGAGCCACA-1.C.txt
bc1d/temp/sparse_matrices/CTAACTTAGAGCCACA-1.G.txt
bc1d/temp/sparse_matrices/CTAACTTAGAGCCACA-1.T.txt
bc1d/temp/sparse_matrices/CTAACTTAGAGCCACA-1.coverage.txt
This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait.
  File "/fast/users/obermayb_c/scratch/miniconda/envs/mgatk/lib/python3.6/site-packages/snakemake/executors/__init__.py", line 544, in handle_job_success
  File "/fast/users/obermayb_c/scratch/miniconda/envs/mgatk/lib/python3.6/site-packages/snakemake/executors/__init__.py", line 231, in handle_job_success
MissingOutputException in line 21 of /fast/work/users/obermayb_c/scratch/miniconda/envs/mgatk/lib/python3.6/site-packages/mgatk/bin/snake/Snakefile.Scatter:
Job completed successfully, but some output files are missing. Missing files after 5 seconds:
bc1d/temp/ready_bam/GCCTAGGCAGTTCGGC-1.qc.bam
bc1d/temp/ready_bam/GCCTAGGCAGTTCGGC-1.qc.bam.bai
bc1d/qc/depth/GCCTAGGCAGTTCGGC-1.depth.txt
bc1d/temp/sparse_matrices/GCCTAGGCAGTTCGGC-1.A.txt
bc1d/temp/sparse_matrices/GCCTAGGCAGTTCGGC-1.C.txt
bc1d/temp/sparse_matrices/GCCTAGGCAGTTCGGC-1.G.txt
bc1d/temp/sparse_matrices/GCCTAGGCAGTTCGGC-1.T.txt
bc1d/temp/sparse_matrices/GCCTAGGCAGTTCGGC-1.coverage.txt
This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait.
  File "/fast/users/obermayb_c/scratch/miniconda/envs/mgatk/lib/python3.6/site-packages/snakemake/executors/__init__.py", line 544, in handle_job_success
  File "/fast/users/obermayb_c/scratch/miniconda/envs/mgatk/lib/python3.6/site-packages/snakemake/executors/__init__.py", line 231, in handle_job_success
Exiting because a job execution failed. Look above for error message
Exiting because a job execution failed. Look above for error message
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /fast/work/groups/cubi/projects/2020-06-29_Starossom_lineage_tracing/2020_07_29_cellranger/.snakemake/log/2020-08-06T084234.060411.snakemake.log

there's another error in bc1.snakemake_gather.log probably caused by the missing files
does that help?
thanks a lot!

my best guess is that you are missing java which is used for doing the PCR deduplication... what does which java yield in your machine? And can you run the command with  --keep-duplicates, which will circumvent the java dependency?

hi, thanks for the tip. I have java, but not the right version
$ which java
~/scratch/miniconda/envs/mgatk/bin/java
$ java -version
openjdk version "1.7.0_91"

I managed to track the original error down to a Picard error:
Exception in thread "main" java.lang.UnsupportedClassVersionError: picard/cmdline/PicardCommandLine : Unsupported major.minor version 52.0

after installing java v1.8.0 I get to the same point as when I run with --keep-duplicates
$ mgatk bcall -i test/test_barcode.bam -n bc1 -o bc1d -bt CB -b test/test_barcodes.txt -z -c 1
Mon Aug 10 14:58:00 CEST 2020: mgatk v0.5.6
Mon Aug 10 14:58:00 CEST 2020: Found bam file: test/test_barcode.bam for genotyping.
Mon Aug 10 14:58:00 CEST 2020: Found file of barcodes to be parsed: test/test_barcodes.txt
Mon Aug 10 14:58:01 CEST 2020: User specified mitochondrial genome matches .bam file
Mon Aug 10 14:58:04 CEST 2020: Finished determining/splitting barcodes for genotyping.
Mon Aug 10 14:58:04 CEST 2020: Genotyping samples with 1 threads
Error in `[<-.data.table`(x, j = name, value = value) :
  RHS of assignment to new column 'sample' is zero length but not empty list(). For new columns the RHS must either be empty list() to create an empty list column, or, have length > 0; e.g. NA_integer_, 0L, etc.
Calls: importMito ... lapply -> FUN -> $<- -> $<-.data.table -> [<-.data.table
Execution halted

the problem now seems to be that the variantFiles (e.g., bc1d/final/bc1.A.txt.gz) don't have a header, and thus importSMs in toRDS.R fails when trying to do
dt$sample <- factor(dt$sample, levels = samplesOrder)

should these files have a header and it wasn't created by makeSM in Snakefile.Gather or should colnames be set in importSMs?
thanks again!
Benedikt

The headers aren’t intended to be there and shouldn’t throw the error. Can you send an ls -lRh of the output file? My guess is that the plain text files are being created but empty
On Aug 10, 2020, at 6:21 AM, Benedikt Obermayer<EMAIL_ADDRESS>wrote:

hi, thanks for the tip. I have java, but not the right version
$ which java
~/scratch/miniconda/envs/mgatk/bin/java
$ java -version
openjdk version "1.7.0_91"
I managed to track the original error down to a Picard error:
Exception in thread "main" java.lang.UnsupportedClassVersionError: picard/cmdline/PicardCommandLine : Unsupported major.minor version 52.0
after installing java v1.8.0 I get to the same point as when I run with --keep-duplicates
$ mgatk bcall -i test/test_barcode.bam -n bc1 -o bc1d -bt CB -b test/test_barcodes.txt -z -c 1
Mon Aug 10 14:58:00 CEST 2020: mgatk v0.5.6
Mon Aug 10 14:58:00 CEST 2020: Found bam file: test/test_barcode.bam for genotyping.
Mon Aug 10 14:58:00 CEST 2020: Found file of barcodes to be parsed: test/test_barcodes.txt
Mon Aug 10 14:58:01 CEST 2020: User specified mitochondrial genome matches .bam file
Mon Aug 10 14:58:04 CEST 2020: Finished determining/splitting barcodes for genotyping.
Mon Aug 10 14:58:04 CEST 2020: Genotyping samples with 1 threads
Error in [<-.data.table(x, j = name, value = value) :
RHS of assignment to new column 'sample' is zero length but not empty list(). For new columns the RHS must either be empty list() to create an empty list column, or, have length > 0; e.g. NA_integer_, 0L, etc.
Calls: importMito ... lapply -> FUN -> $<- -> $<-.data.table -> [<-.data.table
Execution halted
the problem now seems to be that the variantFiles (e.g., bc1d/final/bc1.A.txt.gz) don't have a header, and thus importSMs in toRDS.R fails when trying to do
dt$sample <- factor(dt$sample, levels = samplesOrder)
should these files have a header and it wasn't created by makeSM in Snakefile.Gather or should colnames be set in importSMs?
thanks again!
Benedikt
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/caleblareau/mgatk/issues/21#issuecomment-671350407, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AD32FYNOBMZ44MN6QBRGO3DR77X6FANCNFSM4N4EM3NQ.

I think these files are there and not empty:
$ ls -hlR bc1d/final
bc1d/final:
total 1.6M
-rw-r--r-- 1 obermayb_c hpc-ag-cubi  97K Aug 10 14:58 bc1.A.txt.gz
-rw-r--r-- 1 obermayb_c hpc-ag-cubi 194K Aug 10 14:58 bc1.coverage.txt.gz
-rw-r--r-- 1 obermayb_c hpc-ag-cubi  96K Aug 10 14:58 bc1.C.txt.gz
-rw-r--r-- 1 obermayb_c hpc-ag-cubi   76 Aug 10 14:58 bc1.depthTable.txt
-rw-r--r-- 1 obermayb_c hpc-ag-cubi  50K Aug 10 14:58 bc1.G.txt.gz
-rw-r--r-- 1 obermayb_c hpc-ag-cubi  77K Aug 10 14:58 bc1.T.txt.gz
-rw-r--r-- 1 obermayb_c hpc-ag-cubi 119K Aug 10 14:58 chrM_refAllele.txt
$ zcat bc1d/final/bc1.A.txt.gz | head -n5
2,CTAACTTAGAGCCACA-1,41,52
5,CTAACTTAGAGCCACA-1,47,60
7,CTAACTTAGAGCCACA-1,47,62
13,CTAACTTAGAGCCACA-1,55,72
16,CTAACTTAGAGCCACA-1,59,74

but I don't see how dt$sample <- factor(dt$sample, levels = samplesOrder) can work without defining column names somewhere (in toRDS.R)
sorry for causing so much trouble and thanks a lot for your help!

@bobermayer no! this is actually a bug on my part... thanks for digging through it and finding it!
You're totally right-- the way that I have that line definite isn't right based on the missing header. It's interesting that on my machine, I get a different outcome:
> head(dt)
   V1                 V2 V3 V4
1:  2 ATTGTCTTCCCACTAC-1 10  7
2:  5 ATTGTCTTCCCACTAC-1 10  7
3:  7 ATTGTCTTCCCACTAC-1 10  8
4: 13 ATTGTCTTCCCACTAC-1 11 11
5: 16 ATTGTCTTCCCACTAC-1 11 11
6: 21 ATTGTCTTCCCACTAC-1 14 13
>  dt$sample <- factor(dt$sample, levels = samplesOrder)
> head(dt)
   V1                 V2 V3 V4 sample
1:  2 ATTGTCTTCCCACTAC-1 10  7   <NA>
2:  5 ATTGTCTTCCCACTAC-1 10  7   <NA>
3:  7 ATTGTCTTCCCACTAC-1 10  8   <NA>
4: 13 ATTGTCTTCCCACTAC-1 11 11   <NA>
5: 16 ATTGTCTTCCCACTAC-1 11 11   <NA>
6: 21 ATTGTCTTCCCACTAC-1 14 13   <NA>

It's interesting that your machine throws this as an error but mine didn't. Regardless, this needs to be fixed. I'll bump the version now. Thank you!

@bobermayer try running v0.5.8 (available via PyPi) now?

yes, great, now it works (took a while because I had a very old version of R in my conda environment and I had to fix a few dependencies there).
thanks a lot!

Hello,
I'm getting the same error on the latest version.
  Improper folder specification; file missing / extra file present. See documentation
Calls: importMito -> checkGrep```
I've also tried running with marking duplicates and still end up at the same place.
The output directory looks like this:
```final:
chrM_refAllele.txt

logs:
base.mgatk.log  filterlogs  mgatk.parameters.txt  mgatk.snakemake_tenx.log  rmdupslogs

qc:
quality```

Can you cat the base.mgatk.log and the mgatk.snakemake_tenx.log file?
On Sep 16, 2020, at 4:01 PM, Chang Kim <notifications<EMAIL_ADDRESS>wrote:
Hello,
I'm getting the same error on the latest version.
Improper folder specification; file missing / extra file present. See documentation
Calls: importMito -> checkGrep```
I've also tried running with marking duplicates and still end up at the same place.
The output directory looks like this:
chrM_refAllele.txt

logs:
base.mgatk.log  filterlogs  mgatk.parameters.txt  mgatk.snakemake_tenx.log  rmdupslogs

qc:
quality```

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub<https://github.com/caleblareau/mgatk/issues/21#issuecomment-693710168>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AD32FYMRXENLCBQPARCL4XLSGE7U7ANCNFSM4N4EM3NQ>.

Actually from #22 using --snake-stdout solved the issue.


Add deserialization for TEnum and binary

Fixes the broken tests.

Changes Unknown when pulling 5b3695b8bcf253b71c9c7790f6172780c09f8c8c on pguillory:deserialization2 into ** on pinterest:pure_elixir_implementation**.

Changes Unknown when pulling 5b3695b8bcf253b71c9c7790f6172780c09f8c8c on pguillory:deserialization2 into ** on pinterest:pure_elixir_implementation**.

Gonna go ahead and merge this to get tests working again.


Not able to use object method as a callback function for popups and timers

After using an object method funcref, the object type is not removed from the type stack.
Remove the object type from the stack after getting the funcref.
While addressing this issue, I realized that the ISN_DEFEROBJ instruction introduced by
patch 9.0.1250 is no longer needed as the partial now has a pointer to the object.
The ISN_DEFER instruction can also handle deferring object methods.
So removed ISN_DEFEROBJ instruction.

thanks!


az network vnet show (required parameters)

Hi Team,
I think the command az-network-vnet-show needs two required parameters:

--name -n
--resource-group -g

https://learn.microsoft.com/en-us/cli/azure/network/vnet?view=azure-cli-latest#az-network-vnet-show
Regards,
Hussein

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: bc99d572-abdb-ab6b-7a9a-c6951a1c1466
Version Independent ID: 0a50e966-cbc8-8960-c01f-e117cac842d4
Content: az network vnet
Content Source: latest/docs-ref-autogen/network/vnet.yml
Service: virtual-network
GitHub Login: @rloutlaw
Microsoft Alias: routlaw

@necusjz for awareness

@halkazwini -n and -g are optional when --ids is explicitly specified.


Toggle the panel bar

@jenstornell wrote:

Now I know why I always turn off the bar in WordPress. It's often in the way at the top and adds too much bloat in the HTML.
I still belive in a panel bar if it's flexible as hell.
Make it possible to hide it whenever you want with a "+" toggle-button or so. When hidden, just leave a fixed toggle-button to be able to show it when ever needed.
Make use of panelbar::hide(). When using this function the panel can be hidden on load. Just a toggle-button visible. When using panelbar:show() it can be visible on load.

Random thought: The panel bar would be visible every time the page loads new, @jenstornell? Cause I'm not sure if I'd really want to store the toggle information in a cookie or so. Ideas?

My idea here was simply:
Visible on load:
panelbar::show(args)

Hidden on load:
panelbar::hidden(args)

If my choice is to use panelbar::hidden() I chose to not see the panel at first site, just a toggle-switch.
No need for cookie or storing stuff.

Okay, but if you use
panelbar:show()

The bar would be visible on load, could be hidden by pressing the switch, but visible again as soon as the page reloads / loads another page?

Yes.


Ability to insert the computer name at the end of subject line for email notifications

Add the ability to insert the computer name at the end of subject line for email notifications, per win-acme install.  Now that we have the ability to customize the computer name per win-acme install, in enhancement request "Ability to customize the computer name used for email notifications #1490".  Now that the computer name can be customized in settings.json, I think it would be useful to add the ability to insert the computer name at the end of the subject line, so that someone can know what machine that cert renewal originated from easily (just by viewing the subject line and not needing to read the whole email text).
Examples:
settings.json has the ComputerName variable defined, and is not the default "null" value:
Variable: "ComputerName": "webservices.domain.com"
Subject: Certificate renewal completed [webservices.domain.com]
Subject: Error processing certificate renewal [webservices.domain.com]
Subject: Test notification [webservices.domain.com]
If the "ComputerName" variable is custom defined (and is not the default "null" value) then insert the computer name at the end of the subject for renewal completed, renewal failed, and Test notifications, in brackets or separated by a hyphen.  If the "ComputerName" variable has not been defined (and is the default "null" value") then ignore this and do not insert the computer name in the subject.  This enhancement request could supplement the related email notification enhancement request "Ability to customize the end portion of the subject line for email notifications #1498".
Thank you.

As per 2.1.7 the computer name is added at the end of the subject


Add version to existence check, add source parameter

If version parameter is supplied to cChocoPackageInstall, check for existence of the package includes
version test.
Added source parameter to .mof.
If source parameter is supplied to cChocoPackageInstall, choco source command is executed to get
package from specified source.

Looks good, I'll give this a test and all being well merge it in.


Force layouts don't seem to work when using ComboCombined

Describe the bug
I am trying to display a graph that, on page load, only shows combos (by default, all combos are collapsed). Upon expanding the combos (some of them nested several layers deep), it should display the nodes and their edges.
However on page load the goal is to only show collapsed combos, and have it act similarly to the Christmas Bubbles Example. After several days of trying to debug why I couldn't seem to get this working, I discovered that it appears that when using ComboCombined or ComboForce as the layout, none of the force directed layouts work properly on the combos or nodes displayed. Even when preventOverlap is enabled, none of the nodes or combos seem to have an effect on pushing the others around it.
As an example, I took the Christmas Bubbles examples given in the documentation, and modified it so that it uses the comboCombined layout, and placed its force layout into the outerLayout configuration (I also added the drag-node mode to make dragging work). With only that change, the "force" part of the force layout no longer seems to have an effect, and nodes are neither drawn closer to each other, nor pushed away when dragging one node on top of another.
Your Example Website or App
https://codesandbox.io/s/cool-worker-y9mmtg
Steps to Reproduce the Bug or Issue
As above
Expected behavior
When dragging nodes or combos that have an outerLayout of force, it should act the same as the ChristmasBubbles example, and push other nodes/combos away when dragging one node/combo over another.
Screenshots or Videos
No response
Platform

OS: [e.g. macOS, Windows, Linux]
Browser: [e.g. Chrome, Safari, Firefox]
Version: [e.g. 91.1]

Additional context
No response

Could you give me the demo with your combo data? I saw the data in the demo now is from Christmas bubbles

@Yanyan-Wang Here's a demo with similar data to what I'm using: https://codesandbox.io/s/modest-http-pwttr9


Using X6 connectors and routers in G6 dagre layout

G6 Version: 4.1.3

Where can I find examples on using X6 edge connectors and routers in G6 dagre layout?

They are separated libraries, there are no examples for combining usage. All the demos are on the official websites
https://g6.antv.vision/en/examples/gallery
https://x6.antv.vision/en/examples/gallery/

They are separated libraries, there are no examples for combining usage. All the demos are on the official websites
https://g6.antv.vision/en/examples/gallery
https://x6.antv.vision/en/examples/gallery/


Animation speed

Hi, is there a possibility to change the animation speed.

setSuggestionsAnimDuration(long duration)


client example not handling closed connection

The client example is not handling the case where server suddenly drops the connection, since it's blocked waiting for keyboard input.
Also there seems to be no reason that input is being run on a background thread, only to then await on the result.
There should be two tasks running in parallel: one to accept and process commands, and another to loop on get_message() and queue the results.

Taking a step back, it's odd that the API defers the closed connection event until the next call to get_message().   The exception should be surfaced from the API as soon as it happens so that the nursery passed to connect() will get cancelled.  This is another argument for using start() to invoke connect() and mentioned in #20.
In fact the API should be promoting use of an async context manager:
async with trio_websocket.open_connection(host, port) as connection:
    try:
        ...
    except ConnectionClosed:
        pass

I think raising at the WebSocket I/O call is the most user-friendly way to do this (see the database example in this post). Cancelling the nursery is pretty unfriendly because the caller has to handle ClosedConnection and Cancelled correctly, and the code path that catches Cancelled cannot get the WebSocket's close code.
Maybe there needs to be an optional cancel scope for use cases like the example client, where we would prefer for our handler to be cancelled.

I've written up a quick implementation with an optional cancel_scope argument for client connections. This scope is cancelled when the connection closes. I updated the example client. Here's a sample run where the server closed the connection while the client was waiting for the user to enter a command.
(venv) $ python examples/client.py ws://localhost:8000/asdf
DEBUG:root:Connecting to WebSocket…
DEBUG:trio-websocket:Connecting to ws://localhost:8000/asdf?
DEBUG:trio-websocket:conn#0 sending 158 bytes
DEBUG:trio-websocket:conn#0 received 110 bytes
DEBUG:trio-websocket:conn#0 received event: ConnectionEstablished
DEBUG:root:Connected!
cmd> DEBUG:trio-websocket:conn#0 received zero bytes (connection closed)
DEBUG:trio-websocket:conn#0 websocket closed <ConnectionClosed <CloseReason code=1006 name=ABNORMAL_CLOSURE reason=None>>
DEBUG:trio-websocket:conn#0 reader task finished
DEBUG:root:Connection closed

I don't love this implementation. It doesn't provide an easy way for the user to know that the server closed the connection. It also adds another argument to the WebSocketConnection constructor and the client factory functions. Definitely welcoming any feedback here...
Other possible implementations:

Add a [synchronous] callback function that is called when the connection closes. This still pollutes the arguments for the constructor/factories, but it is more general than cancellation and the close reason could be passed to the callback.
Add an async def wait_closed() function that blocks until the connection closes and then returns the close reason. This does not pollute the arguments for the constructor/factories, is more general than cancellation, and provides the close reason to the caller. I think this is my favorite approach but I worry that it is over-engineered, especially since I don't have any use cases for this "handle connection closed immediately" behavior other than the example client.

Thoughts?

Passing in an optional Trio.Event() would be more general than a cancel scope, while avoiding callbacks.  I've got a decently sized codebase built from day-one on trio and there's not a single callback to be found.
I think the "handle closed connection immediately" use case is real.  For example, what if the client only sends a command occasionally, but at those times needs a response as fast as possible.  It's important to keep a connection active, reconnecting as needed.
However I'm not convinced any new API is necessary, since we already get a signal when the connection closes, by way of the open_websocket_url context manager exiting.  Rather it may just be an issue with the structure of this example.
I tried to address this simply in #52.  But it breaks the regular send and receive case, I can't figure out why.

I'm not sure how to handle this kind of automagical cancel-on-disconnect situation. (It also comes up on the server side – if a connection drops, should you immediately cancel the connection handler? In an HTTP server, should you pre-emptively cancel a request handler if the client disappears and won't be able to receive the response?)
In general in Trio the bias is toward delivering notifications by someone doing await wait_for_event(), not via callbacks or cancellation or whatever. So my gut feeling is that if you want to notice when the connection drops, maybe have a task to listen for that or something?
In this case, wouldn't it make more sense anyway to have a background task that listens for incoming messages and prints them? and if it gets an EOF, then it could raise SystemExit or something?

(ignore my previous comment about a signal already being available-- due to confusion from outdated dependencies)
Putting this particular example aside, aggressive reconnect seems like something people will want and, if not supported automatically by the library, there should at least be an example users can look at.
Monitoring in a separate task sounds reasonable, but at a minimum WebsocketConnection should expose a trio.Event for closed so that users don't have to poll is_closed().  (This is along the lines of wait_closed() proposed by @mehaase but more general.)

What the difference between await conn.close_event.wait() and await conn.wait_closed()?

Not much-- I was thinking it might be easier to pass around event or there may be utils which take event.  But just using thunks is probably more common.  And the event is more dangerous since it's mutable property.

I'm not 100% sure wait_closed is even a good idea actually, versus using receive.
The thing is, underneath, the only way to actually watch for the socket being closed is to read events off of it. But since you don't want to buffer an arbitrary number of events, if some events arrive and the user hasn't called receive, then eventually we'll have to stop reading from the socket until they do call receive. Which means wait_closed is unreliable unless some other task is regularly calling receive. But if a task is regularly calling receive, then they'll get notified of closure, so the wait_closed seems unnecessary.

@njsmith: I'm not 100% sure wait_closed is even a good idea actually, versus using receive.

This point about userspace buffering is a very convincing argument in favor of waiting for a ConnectionClosed exception over any variant of wait_closed().

@belm0 I think the "handle closed connection immediately" use case is real. For example, what if the client only sends a command occasionally, but at those times needs a response as fast as possible. It's important to keep a connection active, reconnecting as needed.

Yes, I see there are two scenarios here. One is the "userspace buffer is full" scenario that Nathaniel discussed. The other is a "no activity on the socket" scenario that you brought up. In your scenario, I think we should recommend users to create a heartbeat task that sends pings. We can provide a recipe for this in the docs.
  async def heartbeat(ws):
    while True:
      await trio.sleep(N)
      await ws.ping()

The ping() method will raise ConnectionClosed, so the user can start the hearbeat task in the same nursery as the handler and when the connection closes it will cancel the nursery.
I'm not sure if the current ping() is ideal: it sends a ping frame and then returns. So the "recipe" above doesn't actually check if any pong frames are received, but it would at least detected a dropped connection.
Maybe ping() should be modified to wait for a corresponding pong()? The WebSocket RFC allows a payload with a ping frame, so we could send a random nonce and then continue when we receive a pong frame with the same nonce.

@njsmith: In this case, wouldn't it make more sense anyway to have a background task that listens for incoming messages and prints them? and if it gets an EOF, then it could raise SystemExit or something?

Yes, this is exactly how it should be. I'll create a new PR with this design.

Maybe ping() should be modified to wait for a corresponding pong()

One subtlety to watch out for here:

If an endpoint receives a Ping frame and has not yet sent Pong frame(s) in response to previous Ping frame(s), the endpoint MAY elect to send a Pong frame for only the most recently processed Ping frame.

So if we did do this, we'd want to make sure that receiving a pong for a later ping would also wake up earlier pings. (Probably not too hard, but it would require that the ping frames be unique and that we keep a central list of outstanding pings.)

Oh, wow, good catch in the spec. I think that completely blows up my desire for each call to ping() to wait for corresponding pong, since the remote endpoint isn't required to always send one. A couple other ideas:

A pong wakes up all tasks that are waiting on ping(). If user cares about matching each ping to a pong, then they are responsible for only having one ping/pong in flight at a time.
Place a counter/timestamp in each ping: each pong wakes up pings less than or equal to the pong's counter/timestamp.

Given that users may want to use the ping payload for their own purposes, I lean towards #1. This would require minimal changes to the current implementation and would mostly consist of a recipe in the docs, to go along with the heartbeat recipe mentioned above.

I checked how aaugustin's websockets library handles the ping thing, and I wasn't disappointed. It looks like its approach is:

Keep an OrderedDict of outstanding pings
When sending a ping, assign a random payload that's not currently outstanding
Or, the user can provide a custom payload, but if it duplicates one of the outstanding payloads, raise an error
ping waits for a reply (by returning a Future; since it's asyncio you have the option of ignoring this)
When a pong is received, use the OrderedDict to tell whether it's a solicited pong, and if so then alert the appropriate call to ping + all previous calls

My inclination would be to copy this design, except make ping an async function so that waiting for the pong is mandatory. If you want to do a fire-and-forget ping, that's what pong is for...

Here's the source i was looking at: https://github.com/aaugustin/websockets/blob/master/src/websockets/protocol.py
In particular check out the methods ping, pong, and read_data_frame (which handles incoming pongs).

I've submitted PR #58 that mimics aaugustin's design. I'll leave 56 and 58 open for now so we can compare the two approaches.


orml-claims

orml-claims

Types

trait OnClaim<AccountId, P>

fn on_claim(who: &AccountId, payload: P) -> DispatchError

This should fail on duplicated claim

Config

type OnClaim: OnClaim
type UpdateTrieRootOrigin: EnsureOrigin

Storage

TrieRoot: Hash

Calls

fn set_trie_root(origin, new_root: Hash)

ensure UpdateTrieRootOrigin
update TrieRoot

fn claim(origin, to: AccountId, payload: Payload, proof: Vec<u8>)

ensure signed
verify proof
OnClaim::on_claim(to, payload)

Note

This is intented to be used to store crowdloan rewards
Proxy / Multisig accounts can send Transact XCM on relaychain to invoke claim
Need to make this a unsigned transaction so people can claim it without paying fee

Sharing a bit more on what Centrifuge has been working on here: https://centrifuge.hackmd.io/NPmnf3a2R3SAN75nA-QSRw

Is the intention that the trie root matches the format of the relaychain crowdloan child trie?

Is the intention that the trie root matches the format of the relaychain crowdloan child trie?

Yes. To keep it simple. It should be possible to abstract it away but unless someone request it, I want to avoid unnecessary features to keep thing simple.

I would suggest we call it orml-crowdloan-claims to make it a bit more specific :)

This is designed to be used with crowdloan but I think it can also be repurposed for other use cases such as airdrops.

@xlc have you started your own implementation of this? Our progress is here: https://github.com/centrifuge/centrifuge-chain/pull/342

We have not started working on this. Depends on how much time we have before crowdloan, we may just distribute rewards via batch call.

We won't be working on this.


Pickup (f) doesn't work in compatibility mode

When in "old" mode pickup_ticks is never set, so the tas doesn't know it is in pickup state. However it also doesn't progress the step counter forward, so the character just stands there.

There is not compatibility mode for pick up. Currently the user is supposed to align the run with the new pickup feature, but I might do something to fix it.

I can confirm that the state does go into compatibility version of the doStep.
Getting a good solution is likely impossible, given how different the old and the new version is. So spend an appropriate amount of time


error: cannot add path '/nix/store/zq4iasx6689adcbawyjls8771gx00rv0-disko' because it lacks a signature by a trusted key

hello,
I am running into a new problem when running the basic command to install on a freshly booted remote bare metal machine:
[tru@nixos:~/nixos-anywhere]$ nix run github:numtide/nixos-anywhere --no-require-sigs --  --flake '.#nixos-efi-toy'<EMAIL_ADDRESS>...
### Gathering machine facts ###
Warning: Permanently added '<IP_ADDRESS>' (ED25519) to the list of known hosts.
### Formatting hard drive with disko ###
Warning: Permanently added '<IP_ADDRESS>' (ED25519) to the list of known hosts.
[1/0/1 copied (0.0/0.0 MiB)] error: cannot add path '/nix/store/zq4iasx6689adcbawyjls8771gx00rv0-disko' because it lacks a signature by a trusted key
error: unexpected end-of-file

I haven't seen this "it lacks a signature by a trusted key" message a few days ago.
Thanks
Tru

Hi,
transient error? it is not happeing this morning with the exact same command... so closing the issue with nothing to do.


Add dedicated DroneCI env compatibility layer

for more info see https://github.com/drone-plugins/drone-plugin-lib/pull/37/files
and c79d49c862ad800f30abcd4fc9b543f99a8e70f9

Maybe, we can look if we can extend this later to other CI systems like GH Actions, Circle CI or whatever else

as long as there plugins are handled via environment vars and do run in a docker container ... yes we can


Need to write test cases to cover HTTP/2 scenarios in transport

$subject is needed since right now we have only an abysmal amount of tests (well around 7)in transport for HTTP/2.

Need to verify the following 2 basic scenarios with an h2 server. (Identified when MicroGateway testing)

The response for an h1 request is always h1.
The response for an h2 request is always h2.


Use Github repo connector for configurations

Solves #542

Codecov Report
Attention: 1314 lines in your changes are missing coverage. Please review.

Comparison is base (5a0e6b3) 36.81% compared to head (cd87380) 53.36%.
Report is 372 commits behind head on main.

Additional details and impacted files
@@             Coverage Diff             @@
##             main     #543       +/-   ##
===========================================
+ Coverage   36.81%   53.36%   +16.54%
===========================================
  Files          61       69        +8
  Lines        5421     6673     +1252
===========================================
+ Hits         1996     3561     +1565
+ Misses       3425     3112      -313

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

@xgui3783, I am also considering refactoring GitlabConnector. Specifically, how we handle archive mode.


Update to use CRD

Just wonder if you have any plan to update the code to use CRD instead of TPR since TPR will be deprecated soon-ish.
More info: https://kubernetes.io/docs/tasks/access-kubernetes-api/migrate-third-party-resource/

Just updated to k8s 1.8 and believe I'm hitting this issue:
error: unable to recognize "ip-pool.yml": no matches for ipcontroller.ext/, Kind=IpClaimPool
any updates on migrating to CRD?

I am also interested, but does not seem to work with k8s 1.8.

Hi folks
Looks like it shouldn't be hard to use TPR instead of CRD. Will try to do it this week


Use proper SDK and command-line tools on OS X 10.11

Some packages do not compile on OS X 10.11 (e.g., #9329). I believe this is because the 10.9 SDK and command-line tools no longer work reliably on 10.11. Replacing the 10.9 SDK and command-line tools with the 10.11 versions fixes the problem in my environment.
I don't know Nix well enough to provide a pull request for a proper fix that works for 10.11 and earlier versions, but for someone more knowledgeable, I believe the relevant Software Update catalog for the 10.11 tools is here:
https://swscan.apple.com/content/catalogs/others/index-10.11-1.sucatalog

BTW, once you do this, the next thing that happens is that, once clang is built, you will run into the linker problems again as described in #2208 (can't find -lSystem). So far, the only way I've been able to get around that is to add:

export NIX_LDFLAGS_AFTER+=" -L/usr/lib"
export NIX_ENFORCE_PURITY=

to the Darwin stage 3 stdenv prehook.
I've worked on it for 2 days and am really perplexed why the linker needs this when bootstrapped with the 10.11 command-line tools, but I don't understand the clang/ld wrapper script very well.

Actually, the built linker doesn't look in any of the standard locations, so you need to tell it about the system framework paths, as well:

export NIX_LDFLAGS_AFTER+=" -L/usr/lib -F/Library/Frameworks -F/System/Library/Frameworks"
export NIX_ENFORCE_PURITY=

nixpkgs should be using the 10.11 SDK on 10.11, just like it uses the 10.9 SDK on 10.9, and 10.10 on 10.10.
We run
/usr/bin/xcrun --sdk macosx"$(/usr/bin/sw_vers -productVersion | /usr/bin/cut -d. -f1,2)" --show-sdk-path 2> /dev/null || echo /

to find the correct SDK, falling back to / in case XCode isn't present. Can you paste the output of that command? I don't have 10.11.

It's finding the 10.11 SDK correctly. Here are some relevant environment variables from a nix-shell:

NIX_LDFLAGS_AFTER= -L/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/usr/lib -L/usr/lib -F/Library/Frameworks -F/System/Library/Frameworks
SDKROOT=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk
NIX_CFLAGS_COMPILE= --sysroot=/var/empty -idirafter /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/usr/include -F/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/System/Library/Frameworks -Wno-multichar -Wno-deprecated-declarations -isystem /nix/store/bzslh5v1pq5wbj22a31j1cachqpjkwzl-libc++-3.7.0/include

(Note that I added the /usr/lib and Frameworks bits to NIX_LDFLAGS_AFTER in my fork.)

If it's finding the 10.11 SDK, you shouldn't need to add -L/usr/lib, the correct place to look for system libraries is the SDK.
Can you post a concrete error that your NIX_LDFLAGS fixes? Perhaps also the contents of /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/usr/lib?

See #2208 for the kind of errors that it fixes. Without -L/usr/lib the very first 'configure' check that runs after stdenv stage 3 is built (I believe it's perl?) breaks, saying that the compiler can't produce executables because it can't find -lSystem.
I'm afraid I don't have time at the moment to go back to master and rebuild from scratch to give you a specific error. I've already spent way too much time fixing this recently, and I've had to hack in a bunch of other fixes for 10.11 and Xcode 7 just to get to the point of being able to produce a stage 3 stdenv in the first place. They're in my darwin branch on my nixpkgs fork, if you're curious. (I haven't opened a pull request for them as they're just expedient hacks and will need some work from someone more knowledgeable about Nix than I before they're ready to be committed.)
Here are the contents of the 10.11 SDK /usr/lib: https://gist.github.com/dhess/b3a4efed3a2e3c1c042e
As you can see from the listing, libSystem.dylib has been replaced by a .tdb file, which contains some textual information about the dylib and its path, but not the modules. This is why the ldflags are needed to point at the actual dylib and Framework locations.

The .tdb stuff is very strange, in the current version of XCode we have actual .dylibs in the SDKs. No wonder things are majorly broken on 10.11..
Based on https://forums.developer.apple.com/thread/4572 it seems that the .tdb files aren't actually supported by xcode (and presumably clang) yet. @pikajude do you know anything about this?

I've had no problems linking with the Xcode 7 command-line clang, so either it knows about the .tdb files in the SDK or it just looks in the system default areas. But even llvm-3.7 clang (released early September) doesn't know how to handle the .tdb files, so unless there's a 3.7.1 with support for them very soon, it seems pretty clear that Nix won't be able to use the SDK as a source for dylibs or Frameworks on 10.11.

@dhess I've tried building with your changes to the stage3 prehook, but it's still not finding System, e.g., when building bzip2. Seems like it's not picking up the changes at all:
...
clang -Wall -Winline -O2 -g -D_FILE_OFFSET_BITS=64 -c decompress.c
clang -Wall -Winline -O2 -g -D_FILE_OFFSET_BITS=64 -c bzlib.c
rm -f libbz2.a
ar cq libbz2.a blocksort.o huffman.o crctable.o randtable.o compress.o decompress.o bzlib.o
ranlib libbz2.a
clang -Wall -Winline -O2 -g -D_FILE_OFFSET_BITS=64 -c bzip2.c
clang -Wall -Winline -O2 -g -D_FILE_OFFSET_BITS=64  -o bzip2 bzip2.o -L. -lbz2
ld: library not found for -lSystem
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Try my fork, the 'elcap' branch. Works fine for me.

I cannot get it to work. Restarted over, from master, with the changes mentioned above.
Ran nix-build pkgs/top-level/all-packages.nix -K -A bzip2
<snip>
clang -Wall -Winline -O2 -g -D_FILE_OFFSET_BITS=64  -L/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/usr/lib -L/usr/lib -F/Library/Frameworks -F/System/Library/Frameworks -o bzip2 bzip2.o -L. -lbz2
ld: library not found for -lSystem

Running from the build dir that's kept yields (adding -v):
Apple LLVM version 6.0 (clang-600.0.57) (based on LLVM 3.5svn)
Target: x86_64-apple-darwin15.0.0
Thread model: posix
 "/nix/store/0ygdzy2h6vvd81kxiqll678lbg9bl30a-CLTools_Executables_OSX109/Library/Developer/CommandLineTools/usr/bin/ld" -demangle -dynamic -arch x86_64 -macosx_version_min 10.7.0 -syslibroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk -o bzip2 -lcrt1.10.6.o -L/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/usr/lib -L/usr/lib -L. bzip2.o -lbz2 -lSystem /nix/store/0ygdzy2h6vvd81kxiqll678lbg9bl30a-CLTools_Executables_OSX109/Library/Developer/CommandLineTools/usr/bin/../lib/clang/6.0/lib/darwin/libclang_rt.osx.a -F/Library/Frameworks -F/System/Library/Frameworks
ld: library not found for -lSystem
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Removing the syslibroot option finally makes it build.
/nix/store/0ygdzy2h6vvd81kxiqll678lbg9bl30a-CLTools_Executables_OSX109/Library/Developer/CommandLineTools/usr/bin/ld -demangle -dynamic -arch x86_64 -macosx_version_min 10.7.0 -o bzip2 -lcrt1.10.6.o -L/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/usr/lib -L/usr/lib -L. bzip2.o -lbz2 -lSystem /nix/store/0ygdzy2h6vvd81kxiqll678lbg9bl30a-CLTools_Executables_OSX109/Library/Developer/CommandLineTools/usr/bin/../lib/clang/6.0/lib/darwin/libclang_rt.osx.a -F/Library/Frameworks -F/System/Library/Frameworks

At this point I've not really an idea how to remove that when building the stdenv. When syslibroot is defined, it's prepended to the other search dirs, so it can only find the .tbd files and not the actual .dylib files.

Could we close this?
For 10.11/Xcode 7 the new default stdenv should address some of these issues.
More information here: https://nixos.org/wiki/Nix_on_OS_X

Which should address these issues, the pure stdenv, or the impure stdenv?

The new default, pure stdenv.

I'll try it on my 10.11 system and let you know.

Before I do so, a) is it safe to enable the sandbox now? and b) I assume I need to do this with nixpkgs from git as it's not in the unstable channel yet, yes?

I would use Nix HEAD:
git clone https://github.com/NixOS/nix
cd nix
nix-env -f release.nix -iA build.x86_64-darwin --option build-use-chroot false

and set
build-use-chroot = true

in /etc/nix/nix.conf
And yes, I would use a clone of nixpkgs, as suggested in the wiki

OK, just for starters, after starting over from a clean slate (removed all traces of nixpkgs from my system, then installed nix-1.10 from the install script on https://nixos.org/nix, Xcode 7.1 and CLI tools are present), building Nix head fails:
checking whether Perl is recent enough... yes
checking for the Perl installation prefix... ${exec_prefix}/lib/perl5/site_perl/5.20.2/darwin-2level
checking for cat... /nix/store/vwyalibj4ml22gcniih6laf2ld37vbi9-coreutils-8.24/bin/cat
checking for tr... /nix/store/vwyalibj4ml22gcniih6laf2ld37vbi9-coreutils-8.24/bin/tr
checking for openssl... /nix/store/83za6xpr2jw6s93fxz4n3hf0fqdn6hcd-openssl-1.0.1p/bin/openssl
checking for pkg-config... /nix/store/d0fnz0f5kkiymiyjxyp0q8qkclibv98h-pkg-config-0.28/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for OPENSSL... yes
checking for BZ2_bzWriteOpen in -lbz2... yes
checking bzlib.h usability... yes
checking bzlib.h presence... yes
checking for bzlib.h... yes
checking for SQLITE3... yes
checking for LIBCURL... yes
checking for SODIUM... yes
checking for LIBLZMA... no
configure: error: Package requirements (liblzma) were not met:

No package 'liblzma' found

Consider adjusting the PKG_CONFIG_PATH environment variable if you
installed software in a non-standard prefix.

Alternatively, you may set the environment variables LIBLZMA_CFLAGS
and LIBLZMA_LIBS to avoid the need to call pkg-config.
See the pkg-config man page for more details.
build time elapsed:  0m0.083s 0m0.068s 0m3.731s 0m2.171s
builder for ‘/nix/store/x7rv5ar8yq5ayqdzl850z9m9gkj37m9l-nix-tarball-1.11pre1234_abcdef.drv’ failed with exit code 1
cannot build derivation ‘/nix/store/c8x6g910wd3vnznr3w566nm54qp8mzlp-nix-1.11pre1234_abcdef.drv’: 1 dependencies couldn't be built
error: build of ‘/nix/store/c8x6g910wd3vnznr3w566nm54qp8mzlp-nix-1.11pre1234_abcdef.drv’ failed

Looks like https://github.com/NixOS/nix/issues/671

I've worked around it, now doing a full build of my standard environment and I'll update you on how it goes.

OK, it's working fine here -- tested with several personal Haskell projects using GHC 7.10.2 from nixpkgs. Thanks for everyone's work on the Darwin pure stdenv! I think you can close this issue now.

Glad to hear it.  I don't have the permissions to close this, but can't you do it as the person who opened it?


Issue: Readme does not tell you what to do if You get Could not get access token: (EMAIL)

ReadMe has these instructions: Reversed API for Google's ImageFX also known as Imagen 3
Login to ImageFx, and then run
python utilities/reverse_imagen3/get_auth.py <variable_name>
This will collect your cookies and store it in .env in the same directory as these files. The variable name is a way of differentiating between different cookie strings, therefore allowing the use of multiple accounts. See the example with run_xfce4.py. Suggest to name your variable name similar to the username you use for your Google account.
This will retrieve the cookie string from .env and use it to get an access token which can be used for generating images for the day. You can expect to generate about 50 images daily before getting a HTTP 429, stopping you from generating more for the day.
python utilities/reverse_imagen3/imagen3.py <variable_name>
image
sudo python run_xfce4.py  (more versions coming soon) will create multiple instances depending on how many variables are found in .env A default of 8 images maximum will be returned per request
If a HTTP 401 is returned with the following message: Request had invalid authentication credentials., it most likely means that the access token has expired. Follow the steps above starting from get_auth.py. For now, you will have to manually remove the string from .env. TODO: automate this
Occasionally a HTTP 400 with message: Request contains an invalid argument is returned. It could mean that inappropriate content has been generated and therefore is not sent back to the client.
What it doesn't tell you is how to resolve "Could not get access token" if you run python utilities/reverse_imagen3/imagen3.py (EMAIL) and in fact HAVE your email and cookie  in the .env.  Please update the Read.me for cross compatibility issues that may arise from people like me.

receipts

good job putting your email out on the internet smh
https://github.com/speckly/sucorn/commit/e3ec7a0b5b0bfd830916cb7cb5d3a41ca57ade72


Non-developer README contribution

I have written a clear, easy-to-comprehend introduction for d3.js. I believe this introduction, free of complex technical terms, will swiftly and thoroughly inform anyone about the use-cases, merit, and advantages, including the celebrated features, of d3.js. Seasoned developers, new programmers, and even nontechnical visitors will benefit from the clear language, exposing d3.js to a broader range of users.

Thanks for the suggestion; however, I prefer the current text.


[sodium fork] Not working with ConnectedBlockTextures

The normal 1.16.x/next works with ConnectedBlockTextures, however your fork seems to cause a crash when joining world.

https://paste.ee/p/EQrdA <-- log

My fork is slightly out of date, I'll fix it now

Thank you so much for fixing this.


[Proposal]: Dropping support for Python 3.7.x legacy

Issue guidelines

[X] I've read the Issue Guidelines and wholeheartedly agree.

Issue Checklist

[X] I have searched open or closed issues and found nothing related to my idea.
[X] I have read the Documentation and it doesn't mention anything about my idea.
[X] To my best knowledge, my idea wouldn't break something for other users.

Describe your Idea
As of 05 Jun 2023, Python 3.7.x legacy has reached its end of support, resulting in the discontinuation of security updates and general widespread availability: https://endoflife.date/python
Use Cases
Consequently, this issue is to inform you that the 3.7.x legacy will no longer be supported in the upcoming version v0.3.2 of vidgear. This implies that starting from version v0.3.2 and onwards, vidgear will no longer function properly on 3.7.x legacy, and any associated bugs pertaining to this legacy version will not be addressed in future releases.
Any other Relevant Information?
https://endoflife.date/python

Additionally, the support for python 3.11 will be officially added in this version. Thank you. 🎉

Successfully replaced python 3.7.x with 3.11.x legacy version in commit  https://github.com/abhiTronix/vidgear/commit/1628685b25fb39a0b59c4af0fd899eedb797b02c 🎉