Batch producer design and features

The block producer is currently two pieces:

The trait BlockProducer
The DefaultBlockBuilder implementation

The trait has a single method:
async fn build_block(&self, batches: Vec<SharedTxBatch>) -> Result<(), BuildBlockError>;

Which is called by TransactionQueue. The current implementation pushes transactions batches on regular intervals.
Things to discuss and improve on the current design:

Back pressure from the block producer. Currently the tx queue pushes batches to the producer on regular intervals without back pressure.

Back pressure here has an unconventional reason. Ideally the block producers will be distributed, meaning the batch proofs are produced by multiple machines, and we could have elastic scaling of the batch provers, such that back pressure would not be necessary.
Even though it is possible to implement auto scaling, e.g. every batch spawns a new machine to perform the proving, making back pressure unnecessary, this would not be a good implementation. There are two issues: 1. it takes time for machines to spawn and be ready to receive work 2. it costs money to run machines. Because of the previous points, instead there should be a pool of provers with a conservative auto scaling policy, meaning there are situations the batch provers may lag behind, causing back pressure.
The unconventional thing is that back pressure is not completely bad, because transactions have to be pooled to be ranked. So there is a balancing act going on allowing some transactions to be pooled, and providing good batching times.

As mentioned above, batch proving will be implemented with a cluster of machines. Currently it is unclear if BlockProducer is an abstraction over the cluster, or a single machine in said cluster.

As mentioned above, batch proving will be implemented with a cluster of machines. Currently it is unclear if BlockProducer is an abstraction over the cluster, or a single machine in said cluster.

IMO it makes sense to abstract over the strategy, local/distributed, instead of a prover instance, a single prover, because the later would need extra code (and maybe even another trait).
If that is the case, then IMO the return type should have Future, so that the distributed case can resolve it to the result and report errors. (Currently it seems this is implemented with a new task

Back pressure from the block producer. Currently the tx queue pushes batches to the producer on regular intervals without back pressure.

If we agree on using the Future as mentioned above, backpressure can be implemented by wrapping the future in an option, the type would be:
async fn build_block(
    &self,
    batches: Vec<SharedTxBatch>,
) -> Option<Future<Result<(), BuildBlockError>>>;

async fn because it allows the inner code to use the gRPC clients wihout much fuss
Option to report back pressure, when None is returned the queue should wait a while before trying again.
Future to represent the work is possible done remotely.

The last thing, is the dependency graph. At the moment the graph is:
graph

   API -->|add_transaction| TransactionQueue;
   TransactionQueue -->|verify_tx| StateView;
   TransactionQueue -->|build_batch| BatchBuilder;
   BatchBuilder -->|build_block| BlockBuilder;
   BlockBuilder -->|apply_block| StateView;

IOW, roughly like a pipeline. (Note: The control is a bit more complicated ref ).
This pipeline-like design, has a few consequences:

The error handling code is on the TransactionQueue, but the successful execution takes the tx and pushes it to the block builder in another task. This makes following the control flow harder, since there are multiple concurrent tasks and queue to keep track of.
It means no one of the components in the stack can keep track of what is in-flight, and what is committed, the consequence is that the state view has to hook into multiple places to track in-flight state.

IMO it should instead be a design like the following:
async fn build_block(
    &self,
    batches: Vec<SharedTxBatch>,
) -> Option<Future<Result<TransactionBatch, BuildBlockError>>>;

That allows the TransactionQueue to in effect take the responsibility of the StateView and we can instead have:
sequenceDiagram
    actor Network
    participant TxQueue
    participant BatchBuilder
    participant BlockBuilder
    participant Store

    Note over Network,Store: New tx
    Network->>TxQueue: add_transaction
    activate TxQueue
    TxQueue->>TxQueue: verify_transaction
    alt is_valid
    TxQueue->>TxQueue: add_tx
    end
    alt has_txs_for_batch
    TxQueue--)BatchBuilder: build_batch
    end
    deactivate TxQueue

    Note over Network,Store: New batch
    BatchBuilder--)TxQueue: batch_done
    activate TxQueue
    TxQueue->>TxQueue: add_batch
    alt has_batches_for_block
    TxQueue--)BlockBuilder: build_block
    end
    deactivate TxQueue

    Note over Network,Store: New block
    BlockBuilder--)TxQueue: block_done
    activate TxQueue
    TxQueue->>Store: apply_block
    TxQueue->>TxQueue: update_inflight
    deactivate TxQueue

    Note over Network,Store: Periodic
    activate TxQueue
    TxQueue->>TxQueue: tick
    loop for each tx group
    TxQueue--)BatchBuilder: build_batch
    end
    alt has_batches
    TxQueue--)BlockBuilder: build_block
    end
    deactivate TxQueue

The trait has a single method:
async fn build_block(&self, batches: Vec<SharedTxBatch>) -> Result<(), BuildBlockError>;

Which is called by TransactionQueue.

This is not accurate (transaction queue pushes transaction batches into the batch builder, and the batch builder pushes batches into the block producer), but since the diagram in https://github.com/0xPolygonMiden/miden-node/issues/191#issuecomment-1911855544 is correct, I'm assuming this is a typo?

IMO it should instead be a design like the following:
async fn build_block(
    &self,
    batches: Vec<SharedTxBatch>,
) -> Option<Future<Result<TransactionBatch, BuildBlockError>>>;

I'm not quite following this part:

Why are we returning a TransactionBatch from build_block()?
Assuming the above is a typo (i.e., TransactionBatch should be Block), why do we return Feature within an Option?

Basically, was pass a set of batches into build_block(), unless there is an error, we should get a block back, and the Feature part is handled organically by async function.

That allows the TransactionQueue to take the responsibility of the StateView

This approach could work and we actually considered it initially (basically to have a single component which manages the whole state with several tasks interacting with it). The current approach tries to follow an "actor model" where there are multiple components each responsible for maintaining their own state. I am not against re-evaluating this though, especially if it leads to a simpler design.
One other general note: we should keep in mind that we are building towards a fully decentralized system (even if it takes us some time to get there). This means that we shouldn't assume that there is a single block builder or even that the block builder will be known ahead of time.
This doesn't have a ton of implications for now, but there are a few. The main one relevant here is that block production times should be relatively regular. Currently, we are aiming 3-second block times (though this can change).

This is not accurate (transaction queue pushes transaction batches into the batch builder, and the batch builder pushes batches into the block producer), but since the diagram in https://github.com/0xPolygonMiden/miden-node/issues/191#issuecomment-1911855544 is correct, I'm assuming this is a typo?

Ah, yes, I meant to write the issue about the batch builder, as in the issue title. Not sure why I pasted the links to the block producer over there.

Why are we returning a TransactionBatch from build_block()?

Because of the typo, it should have been the batch builder.

Assuming the above is a typo (i.e., TransactionBatch should be Block), why do we return Feature within an Option?

It is an Option of a Future.
The Option would signal the batch builder has enough work, and the transaction queue should wait before pushing extra work. This implements back-pressure.
The Future is working like a Promise from the JS world, basically an object that will be resolved once there is a value. It represents the fact that batch proving is asynchronous. Returning a Future should remove the need for spawning a task (maybe), and it signals to the caller that it will receive the results back.

This approach could work and we actually considered it initially (basically to have a single component which manages the whole state with several tasks interacting with it). The current approach tries to follow an "actor model" where there are multiple components each responsible for maintaining their own state. I am not against re-evaluating this though, especially if it leads to a simpler design.

I don't know all the theoretical baggage of the actor model framework. But my main takeaway is the lack of locking primitives for synchronization, which I'm not proposing to use here, so in that regard it should have no downside (as in, this also doesn't deadlock).
Are there any other benefits we are looking for?

The Option would signal the batch builder has enough work, and the transaction queue should wait before pushing extra work. This implements back-pressure.

Could we not accomplish this by adding something like BuildBatchError::Busy variant? Basically, instead of wrapping retuning None in case when the batch builder is busy, we'd return Busy error.
A couple of general comments about Tx queue and batch builder:
The way they are set up currently is very primitive - this was basically implemented to make them work in the simplest possible way. So, we should view these as skeletons of what needs to be there. Here is a rough sketch of the flow we'd like to achieve eventually:
As transactions are submitted, they are added to the transaction queue. The only checks we make here is that transactions should be valid. I think this part works fine.
Assuming the batch builder is not busy, we want to submit transactions to it for batching (if it is busy, ideally, we won't even send a request to batch at all). Here we have 2 considerations:

We want to make batches as "full" as possible (we still need to define what "full" means). For example, we don't want to end up in a situation when we have 1 transaction per batch.
We don't want transactions to be stuck in a queue for too long - so, sometimes, we may be OK with a batch containing 1 transaction.

The batch selection process right now is very primitive: periodically, we grab transactions from the queue, split them up into equal groups, and send them off to the batch builder. We should encapsulate this logic so that it is easy to update without affecting the rest of the system. For example, we could have something like this on the transaction queue:
/// Extracts groups of transactions from the current transaction queue. `ProposedBatch` could be
/// as simple as `Vec<ProvenTransaction>`.
///
/// `num_batches` could come from the batch producer indicating how much capacity it has.
pub fn select_batches(&self, num_batches: usize) -> Vec<ProposedBatch>

Current implementation of this function, would look something like this:
pub fn select_batches(&self, num_batches: usize) -> Vec<Vec<ProvenTransaction>> {
    let txs: Vec<ProvenTransaction> = {
        let mut locked_ready_queue = self.ready_queue.write().await;

        if locked_ready_queue.is_empty() {
            return Vec::new();
        }

        locked_ready_queue.drain(..).collect()
    };

    txs.chunks(self.options.batch_size).map(|txs| txs.to_vec()).to_vec()
}

The implementation of try_build_batches() could then look something like this:
async fn try_build_batches(&self) {
    let num_batches = self.batch_builder.get_capacity();
    if num_batches == 0 {
        return;
    }

    let proposed_batches = self.select_batches(num_batches);

    for proposed_batch in proposed_batches {
        let ready_queue = self.ready_queue.clone();
        let batch_builder = self.batch_builder.clone();

        tokio::spawn(
            async move {
                match batch_builder.build_batch(proposed_batch).await {
                    Ok(_) => {
                        // batch was successfully built, do nothing
                    },
                    Err(e) => {
                        // batch building failed, add txs back at the end of the queue
                        ready_queue.write().await.append(&mut e.into_transactions());
                    },
                }
            }
            .instrument(info_span!(target: COMPONENT, "batch_builder")),
        );
    }
}

Another point is that if we do want to have one central place to maintain global state (e.g., something like TransactionPool), we probably need to change how batch and block builder work. Currently, the build batches/blocks and process the results (e.g., build_batch and build_block do not return anything). But with this approach, the would need to return the result to be processed accordingly. For example, the try_build_batches function would change like so:
async fn try_build_batches(&self) {
    let num_batches = self.batch_builder.get_capacity();
    if num_batches == 0 {
        return;
    }

    let proposed_batches = self.select_batches(num_batches);

    for proposed_batch in proposed_batches {
        let tx_queue = self.tx_queue.clone();
        let batch_queue = self.batch_queue.clone();
        let batch_builder = self.batch_builder.clone();

        tokio::spawn(
            async move {
                match batch_builder.build_batch(proposed_batch).await {
                    Ok(batch) => {
                        // batch was successfully built, add the it to the batch queue
                        batch_queue.write().await.push(batch);
                    },
                    Err(e) => {
                        // batch building failed, add txs back at the end of the queue
                        tx_queue.write().await.append(&mut e.into_transactions());
                    },
                }
            }
            .instrument(info_span!(target: COMPONENT, "batch_builder")),
        );
    }
}

It does work. But IMO this is a more complicated API, for example, this adds the need for the get_capacity you mentioned above. My original idea was to isolate things that can be done synchronously from things that must be done async.

I probably still don't fully understand your proposal. Let's say we have the following method on the BatchBuilder:
async fn build_batch(
    &self,
    transactions: Vec<ProvenTransaction>,
) -> Option<Future<Result<TransactionBatch, BuildBatchError>>>

How would it be called from transaction queue/pool (i.e., how would try_build_batches() look like)?

I probably still don't fully understand your proposal.
How would it be called from transaction queue/pool look like?

The proposal is basically:

txqueue should have handles to the in-flight batches (the Future above).

Needed to propagate cancellations (I'm against fire-and-forget in production)
It allows to push new work once capacity is available (= when a Future resolves).

The back pressure should be in the same task.

Reduces race conditions, because no time is needed to signal back pressure.

I did a few experiments and I no longer think that returning a Future is a good idea. Here is why:

I didn't find a good structure to store the Futures. The code would need a dynamic container, similar to a Vec<Future>, which waits for all the Futures concurrently. FuturesUnordered is the closest to a solution that I found, the problem with this structure is that it doesn't block when it is empty.
My idea was to return the task itself as the Future. The goal was to have all the Futures managed by a single task, so that it would be easy to propagate cancellations to the inner Futures. The problem with returning a Future is that async fn are unnamed types, also the compiler does Sync estimation. The first point means we have to use Box<dyn Future> to store said futures, which is a bit cumbersome. The second point means that the code would be brittle, it is easy to break Sync in an async task and endup in a situation were the code needs to be reworked.

Here is a concrete example. I think changing the type to JoinHandle<Result<Batch, Error>> like you suggested would be better indeed:

Simulation code
use futures::stream::{FuturesUnordered, StreamExt};
use rand::distributions::{Distribution, Uniform};
use std::cmp::min;
use std::ops::RangeInclusive;
use std::sync::Arc;
use std::time::Duration;
use tokio::sync::{mpsc, RwLock};
use tokio::task::JoinHandle;
use tokio::time::{self, Instant};

const BATCH_SIZE: usize = 4;
const BATCH_TIME: u64 = 4000;
const BATCH_MAX: usize = 2;

const TX_FREQUENCY: RangeInclusive<u64> = 700..=2000;
const BATCH_FREQUENCY: RangeInclusive<u64> = 4000..=6500;

type BatchBuilderResult = Result<Batch, ()>;

/// Simulates a proven Transaction
#[derive(Debug)]
pub struct Tx {
    pub id: usize,
}

/// Simulates a proven batch
#[derive(Debug)]
pub struct Batch {
    pub id: usize,
    pub txs: Vec<Tx>,
}

/// Simulates the batch builder
#[derive(Debug)]
struct BatchBuilder {
    batches: Arc<RwLock<usize>>,
    id: usize,
}

/// Type to describe current backpressure state.
#[derive(Debug, PartialEq, Eq, PartialOrd, Ord)]
enum Backpressure {
    Yes,
    No,
}

impl BatchBuilder {
    fn new() -> Self {
        Self {
            batches: Arc::new(RwLock::new(0)),
            id: 0,
        }
    }

    async fn build(&mut self, txs: Vec<Tx>) -> Result<JoinHandle<BatchBuilderResult>, Vec<Tx>> {
        assert!(!txs.is_empty()); // don't waste time on empty batches
        assert!(txs.len() <= BATCH_SIZE);

        {
            let mut batches = self.batches.write().await;

            // simulates all workers being busy
            if *batches >= BATCH_MAX {
                return Err(txs);
            }
            *batches += 1;
        }

        let between = Uniform::from(BATCH_FREQUENCY);
        let mut rng = rand::thread_rng();
        let time = Duration::from_millis(between.sample(&mut rng));
        let id = self.id;
        let lock = self.batches.clone();
        println!("Batches started {id}");
        let handle = tokio::spawn(async move {
            tokio::time::sleep(time).await;
            let mut batches = lock.write().await;
            *batches -= 1;

            Ok(Batch { id, txs })
        });
        self.id += 1;
        Ok(handle)
    }
}

/// Try to create as many full batches as possible, until all transactions are consumed or
/// a backpressure event.
async fn try_full_batches(
    txpool: &mut Vec<Tx>,
    batch_builder: &mut BatchBuilder,
    inflight: &mut FuturesUnordered<JoinHandle<BatchBuilderResult>>,
) -> Backpressure {
    while txpool.len() >= BATCH_SIZE {
        let batch: Vec<_> = txpool.drain(..BATCH_SIZE).collect();

        match batch_builder.build(batch).await {
            Ok(handle) => inflight.push(handle),
            Err(batch) => {
                txpool.extend(batch);
                println!("Batch builder full");
                return Backpressure::Yes;
            }
        }
    }

    Backpressure::No
}

/// Maybe start build a partial batch (one with less than BATCH_SIZE transactions).
///
/// Only used for timer based batches, all other events require a full batch size.
async fn try_partial_batch(
    txpool: &mut Vec<Tx>,
    batch_builder: &mut BatchBuilder,
    inflight: &mut FuturesUnordered<JoinHandle<BatchBuilderResult>>,
) -> Backpressure {
    let size = min(txpool.len(), BATCH_SIZE);
    let batch: Vec<_> = txpool.drain(..size).collect();

    match batch_builder.build(batch).await {
        Ok(handle) => inflight.push(handle),
        Err(batch) => {
            txpool.extend(batch);
            println!("Batch builder full");
            return Backpressure::Yes;
        }
    }

    Backpressure::No
}

async fn txqueue(mut txs: mpsc::Receiver<Tx>) {
    let mut batch_builder = BatchBuilder::new();
    let mut txpool = Vec::new();
    let mut inflight = FuturesUnordered::new();
    let mut backpressure = Backpressure::No;

    // timer used to start a new batch because of latency
    // the timer is reset on every try, regardless of their result
    let batch_time = Duration::from_millis(BATCH_TIME);
    let sleep = time::sleep(batch_time);
    tokio::pin!(sleep);

    loop {
        tokio::select! {
            // disable timer if:
            // - backpressure is enabled
            // - txpool is empty
            () = &mut sleep, if backpressure == Backpressure::No && !txpool.is_empty() => {
                println!("[TXQUEUE] TIME");
                // set backpressure to handle cases the worker pool shrinked
                backpressure = try_partial_batch(&mut txpool, &mut batch_builder, &mut inflight).await;
                sleep.as_mut().reset(Instant::now() + batch_time);
            },
            // never block receiving transactions
            tx = txs.recv() => {
                let tx = tx.expect("recv should not have failed");
                println!("[TXQUEUE] {:?}", tx);
                txpool.push(tx);

                // if there is no backpressure for the time being, trigger a batch right away
                if backpressure == Backpressure::No {
                    backpressure = try_full_batches(&mut txpool, &mut batch_builder, &mut inflight).await;
                    sleep.as_mut().reset(Instant::now() + batch_time);
                }
            }
            // disable if:
            // - the FuturesUnordered is empty, since it resolves right away
            batch = inflight.next(), if !inflight.is_empty() => {
                let batch = batch.expect("task should not have failed").unwrap().unwrap();
                println!("[TXQUEUE] {:?}", batch);
                backpressure = try_full_batches(&mut txpool, &mut batch_builder, &mut inflight).await;
                sleep.as_mut().reset(Instant::now() + batch_time);
            }
        }
    }
}

async fn create_txs(txs: mpsc::Sender<Tx>) {
    let between = Uniform::from(TX_FREQUENCY);
    let mut rng = rand::thread_rng();
    let mut id = 0;

    loop {
        if id % 7 == 6 {
            // trigger time based
            tokio::time::sleep(Duration::from_millis(BATCH_TIME * 2)).await;
        }

        if id % 11 == 10 {
            // overload batch builder
            for _ in 0..(BATCH_SIZE * 2) {
                txs.send(Tx { id }).await.unwrap();
                id += 1;
            }
        } else {
            // regular
            let time = Duration::from_millis(between.sample(&mut rng));
            tokio::time::sleep(time).await;
            txs.send(Tx { id }).await.unwrap();
            id += 1;
        }
    }
}

#[tokio::main]
async fn main() {
    let (sender, receiver) = mpsc::channel(100);

    tokio::join!(txqueue(receiver), create_txs(sender));
}

The mempool rework supersedes this.


Race Condition in MigrationLock

Hi there!
Somewhat related to this PR https://github.com/sergeyzwezdin/Hangfire.Mongo/pull/166, I've found another race condition that exists related to locks.
Since a different collection is used for MigrationLocks, that unique index that was added in https://github.com/sergeyzwezdin/Hangfire.Mongo/pull/166 does not apply and hence no guarantee that the mongo upsert will not insert two locks simultaneously. See https://docs.mongodb.com/manual/reference/method/db.collection.update/#use-unique-indexes for more information.
What ends up happening is that multiple migrations can be run at once, resulting in migration failure. Even when using MigrationStrategy.Drop since a migration is still run under that strategy.
Would recommend that the migration locks use the same collection as the distributed locks to reuse the index there, but could add another index to the current migration lock collection if preferred.

@jondavidford,
Thanks for your input. I use a different approach on the MigrationLock. I use a static Id which all locks share to try to acquire the lock.
Would you agree that this would ensure an atomic operation?
private readonly BsonDocument _migrationIdFilter =
            new BsonDocument("_id", new BsonObjectId("5c351d07197a9bcdba4832fc"));
var update = Builders<MigrationLockDto>
                        .Update
                        .SetOnInsert(_ => _.ExpireAt, lockTimeoutTime);

                    var options = new FindOneAndUpdateOptions<MigrationLockDto>
                    {
                        IsUpsert = true,
                        ReturnDocument = ReturnDocument.Before
                    };

                      var result = _migrationLock.FindOneAndUpdate(_migrationIdFilter, update, options);

Ah, I didn't notice that when reading the code. It would seem that an atomic operation is guaranteed there since the _id should be uniquely indexed.
But based on my observations I believe that somehow multiple migrations are able to run simultaneously. The below stack trace is an example of an error we are seeing happen occasionally when using migration strategy drop. Wondering if you have any other ideas on how that might be possible? Perhaps too short of a migration timeout...? But don't see how it should take longer than 30 seconds when using migration strategy drop.
   at MongoDB.Driver.Core.WireProtocol.CommandUsingCommandMessageWireProtocol`1.ProcessResponse(ConnectionId connectionId, CommandMessage responseMessage)
   at MongoDB.Driver.Core.WireProtocol.CommandUsingCommandMessageWireProtocol`1.Execute(IConnection connection, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.WireProtocol.CommandWireProtocol`1.Execute(IConnection connection, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Servers.Server.ServerChannel.ExecuteProtocol[TResult](IWireProtocol`1 protocol, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Servers.Server.ServerChannel.Command[TResult](ICoreSession session, ReadPreference readPreference, DatabaseNamespace databaseNamespace, BsonDocument command, IEnumerable`1 commandPayloads, IElementNameValidator commandValidator, BsonDocument additionalOptions, Action`1 postWriteAction, CommandResponseHandling responseHandling, IBsonSerializer`1 resultSerializer, MessageEncoderSettings messageEncoderSettings, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.CommandOperationBase`1.ExecuteProtocol(IChannelHandle channel, ICoreSessionHandle session, ReadPreference readPreference, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.CommandOperationBase`1.ExecuteProtocol(IChannelSource channelSource, ICoreSessionHandle session, ReadPreference readPreference, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.WriteCommandOperation`1.Execute(IWriteBinding binding, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.CreateCollectionOperation.Execute(IWriteBinding binding, CancellationToken cancellationToken)
   at MongoDB.Driver.OperationExecutor.ExecuteWriteOperation[TResult](IWriteBinding binding, IWriteOperation`1 operation, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoDatabaseImpl.ExecuteWriteOperation[T](IClientSessionHandle session, IWriteOperation`1 operation, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoDatabaseImpl.CreateCollectionHelper[TDocument](IClientSessionHandle session, String name, CreateCollectionOptions`1 options, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoDatabaseImpl.CreateCollection(IClientSessionHandle session, String name, CreateCollectionOptions options, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoDatabaseImpl.UsingImplicitSession(Action`1 func, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoDatabaseImpl.CreateCollection(String name, CreateCollectionOptions options, CancellationToken cancellationToken)
   at Hangfire.Mongo.Migration.Steps.Version17.AddNotificationsCollection.Execute(IMongoDatabase database, MongoStorageOptions storageOptions, IMongoMigrationContext migrationContext)
   at Hangfire.Mongo.Migration.MongoMigrationRunner.Execute(MongoSchema fromSchema, MongoSchema toSchema)
   --- End of inner exception stack trace ---
   at Hangfire.Mongo.Migration.MongoMigrationRunner.Execute(MongoSchema fromSchema, MongoSchema toSchema)
   at Hangfire.Mongo.Migration.MongoMigrationManager.Migrate()
   at Hangfire.Mongo.Migration.MongoMigrationManager.MigrateIfNeeded(MongoStorageOptions storageOptions, IMongoDatabase database)
   at Hangfire.Mongo.MongoStorage..ctor(MongoClientSettings mongoClientSettings, String databaseName, MongoStorageOptions storageOptions)```

@jondavidford,
Thanks. I'll take a look!

@jondavidford,
Would you be able to reproduce this while enabling logging? you should enable verbosity level INFO.
Thanks!

@jondavidford,
I havent been able to reproduce, but in principle there should be small window as I try to delete the lock object when hangfire is starting up. this could potentially create a window. I addressed it in the referenced link.

Thanks @gottscj, I have not had much luck reproducing this on my end either. That looks promising as a fix. Let's close the issue and I will reopen if I am able to reproduce and provide more evidence.


Consistency and completeness of AWS tables and objects

Is your feature request related to a problem? Please describe.
Steampipe is awesome, it provides a great interface to query AWS resources - something which neither AWS Config, not AWS Resource Explorer give.
While using Steampipe, we found out that:

Some properties are missing from AWS tables
There is no table naming convention that coincide with AWS SDK names. E.g. aws ec2 CLI command is reflect in Steampipe as aws_ec2_instance

Describe the solution you'd like
With respect to the numbers above:

Implement a mechanism to detect what is missing, add the missing properties and object and repeat the process to guarantee consistency and completeness of Steampipe tables data
Document how AWS objects are being translated into Streampipe tables and objects and implement such a naming\translation convention if it is missing

Describe alternatives you've considered
With respect to the numbers above:

Query Steampipe tables names and table schemas and compare them with Boto data-structure
Query Steampipe tables and intelligent guess what is the corresponding table and column that should be use to retrieve specific property

Additional context

Boto API client reference
https://github.com/turbot/steampipe-plugin-aws/issues/1708
https://github.com/turbot/steampipe/discussions/3355

Hey @foolioo , thanks for opening this issue!
We agree that ensuring we're up to date with all properties is a challenge today, and something we don't always keep up to date with in this plugin (and others). I like the idea of adding a mechanism to detect what's missing across the tables. Instead of boto, since we use the AWS SDK Go v2, we could look to compare our columns against the types in that package, e.g., for aws_acm_certificate, look at the CertificateDetail type.
It wouldn't always be a 1:1 mapping, since for many tables, we pull in API data from other APIs as part of our hydrate calls, e.g., tables like aws_s3_bucket can have 10+ additional API calls.
Another approach we've considered is to look at AWS API changes, like those from https://awsapichanges.info/, and try to recognize changes that'd affect our tables, but this also has its own challenges.
For the table naming convention, we generally follow the convention of aws_<service>_<resource>, e.g., aws_s3_bucket, aws_ec2_instance, aws_dax_parameter_group. For the resource name, we generally look at the API/CLI resource name and follow that. There are more details on table naming conventions in https://steampipe.io/docs/develop/standards#naming.
For table names, did you find any that you found inconsistent, or were surprised that they were named that way?

@foolioo We don't have any active ongoing work for automating detection/remediation of missing columns at the moment, but if you have any other thoughts or suggestions around it we're still interested in discussing it, as we'd like to implement it one day.
And I don't believe that familiarity with the AWS CLI/SDK doesn't have any benefit when querying Steampipe. The service and resource names mentioned in the commands, like aws ec2 describe-instances, are used in our table names, like aws_ec2_instance. The reason we don't have a 1:1 mapping for some tables is that often AWS has broken out information about a single resource type across multiple types. For instance, aws ec2 describe-instances has a lot of information about EC2 instances, but if you want to get kernel data, user data, or root instance data, you'd need to use aws ec2 describe-instance-attribute --attribute <attribute here>.
So in terms of service resources, we often have 1 table mapping to each service resource, but for information about each resource, we may pull from multiple APIs for ease of use.

Hey @foolioo, we are closing this issue for now; Please feel free to reopen the issue if you want to share or discuss anything.


Some retrieved taxonomies are not species

When typing pombe in the species - taxon ID input field, several other sources appear (e.g. "Schizosaccharomyces pombe expression vector pDUAL2-HFG1c - 477364"). When selecting this, the following error is displayed: "The selected species does not have a reference assembly."
The api call is the following
https://api.ncbi.nlm.nih.gov/datasets/v2alpha/taxonomy/taxon_suggest/pomb?taxon_resource_filter=TAXON_RESOURCE_FILTER_ALL&tax_rank_filter=higher_taxon
Given that this is not the intended usage, do you think we should limit the search? We can limit to:

species: set tax_rank_filter=species -> Is there any possible drawback?
genomes: set taxon_resource_filter=RESOURCE_FILTER_GENOME" -> Users working with a species without a genome will not get notified and may wonder if SYC has a problem.

I leave the reference to the API docs here: https://www.ncbi.nlm.nih.gov/datasets/docs/v2/api/rest-api/#get-/taxonomy/taxon_suggest/-taxon_query-
Related to #97

This is something I discussed with the NCBI datasets team in the past: https://github.com/ncbi/datasets/issues/317
The issue is that if you filter by species, you would not get E. coli (taxid: 562). I never understood why, so I sticked with all species, that's actually the behaviour you have in the ncbi site for genomes (https://www.ncbi.nlm.nih.gov/datasets/). You can write pombe there and you will get the same results. Below a query where e. coli is not returned.
https://api.ncbi.nlm.nih.gov/datasets/v2alpha/taxonomy/taxon_suggest/coli?taxon_resource_filter=TAXON_RESOURCE_FILTER_ALL&tax_rank_filter=species
However, I just noticed that TAXON_RESOURCE_FILTER_GENOME and species, then E. coli is returned.
https://api.ncbi.nlm.nih.gov/datasets/v2alpha/taxonomy/taxon_suggest/coli?taxon_resource_filter=TAXON_RESOURCE_FILTER_GENOME&tax_rank_filter=species

I thikn I might have seen what the problem is, see my comment in the other repo. I will keep the TAXON_RESOURCE_FILTER_GENOME then.


Microsoft.AspNet.Server.KestrelTests failures on CoreCLR on Linux

Microsoft.AspNet.Server.KestrelTests.EngineTests.DisconnectingClient(testContext: TestServiceContext { AppLifetime = LifetimeNotImplemented { ApplicationStarted = (throws NotImplementedException), ApplicationStopped = (throws NotImplementedException), ApplicationStopping = (throws NotImplementedException) }, ConnectionFilter = null, DateHeaderValueManager = TestDateHeaderValueManager { }, Log = TestKestrelTrace { }, Memory = MemoryPool { Empty = [...] } }) [FAIL]
  System.PlatformNotSupportedException : Operation is not supported on this platform.
  Stack Trace:
       at System.Net.Sockets.SocketPal.Disconnect(Socket socket, SafeCloseSocket handle, Boolean reuseSocket)
       at System.Net.Sockets.Socket.Disconnect(Boolean reuseSocket)
       at Microsoft.AspNet.Server.KestrelTests.EngineTests.<DisconnectingClient>d__19.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
Microsoft.AspNet.Server.KestrelTests.EngineTests.DisconnectingClient(testContext: TestServiceContext { AppLifetime = LifetimeNotImplemented { ApplicationStarted = (throws NotImplementedException), ApplicationStopped = (throws NotImplementedException), ApplicationStopping = (throws NotImplementedException) }, ConnectionFilter = NoOpConnectionFilter { }, DateHeaderValueManager = TestDateHeaderValueManager { }, Log = TestKestrelTrace { }, Memory = MemoryPool { Empty = [...] } }) [FAIL]
  System.PlatformNotSupportedException : Operation is not supported on this platform.
  Stack Trace:
       at System.Net.Sockets.SocketPal.Disconnect(Socket socket, SafeCloseSocket handle, Boolean reuseSocket)
       at System.Net.Sockets.Socket.Disconnect(Boolean reuseSocket)
       at Microsoft.AspNet.Server.KestrelTests.EngineTests.<DisconnectingClient>d__19.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

This affects https://github.com/aspnet/Universe/issues/304.

@halter73 This is weird. Do we have another API that we should be using instead? (To make this work on CoreCLR as well?)

This is the only place we call Socket.Disconnect: https://github.com/aspnet/KestrelHttpServer/blob/8f0d1179ab8cc636febd8a319ba14008d96d037d/test/Microsoft.AspNet.Server.KestrelTests/EngineTests.cs#L398
By simply removing the call, and letting Dispose disconnect the socket, the test passes on both Windows and Linux.

:+1: Let's get that in :)

@halter73 will you be able to check this in with any other PR going in soon?


Error when compiling circuit with Circom: Uninitialized inputs in component cdouble

Hello,
The error occurs in the script ./e2e_test_recursive_proof_circom_verification.sh. Below is the error message:
****COMPILING CIRCUIT****
error[T3001]: Component cdouble is created but not all its inputs are initialized
    ┌─ "/home/plonky2-circom/polymer/plonky2-circom/circom/circuits/goldilocks_ext.circom":149:5
    │
149 │     cdouble[i] = GlExtSquare();
    │     ^^^^^^^^^^^^^^^^^^^^^^^^^^ found here
    │
    = call trace:
      ->VerifyPlonky2Proof
       ->EvalVanishingPoly
        ->EvalGateConstraints
         ->Poseidon12
          ->GlExtExpN

previous errors were found

Here's some context about my setup:

The directories circom, poseidon-permutation-rs, and rapidsnark are at the same level as my project root directory plonky2-circom.
snarkjs is located one level above the project root directory.

It seems like the cdouble component is created without all its inputs being initialized properly.

The problem is solved, simply add these two dummy lines to satisfy circom:
  cdouble[N-1].a[0] <== 0;
  cdouble[N-1].a[1] <== 0;


preset fails to save, just hangs

when i type in "newtest" and hit the save button, it seems to hang

and i see this message in the log

the preset is never created.  if i exit the dialog and come back later it's not listed.

This is less of a bug and more of an issue with local storage being out of space.
The problem here is that if I make the setting a world setting then the presets only exist in one single world, so they are a client setting. I will look to see if I can resolve this another way. Maybe create a local database to store the values in. Though that may have issues with certain hosting environments


README woefully out of date

I think we should focus on doltpy.core over doltpy.etl to get people eased into the interface. At the start, it's more likely that people will come for simple shit like:

I want to read a table into a dataframe
I want to import a CSV
I want to run a SQL query and get a dictionary of the results

It's much less likely that they'll want a full service importer that they can integrate with Airflow easily. We should ease them up to that.

The README is actually not out of date, it just emphasizes shims as the entry point, rather than the API. I agree, but I actually want to figure out how to generate good docs from the docstrings, and use that, and then make the README a copy of what we put up as a tutorial/getting started on the docs site.

As per my above comment we now have a brief up to date README that points folks at our docs site where more detail is warranted.


Software license

What software license is this being released under?  I'm not seeing any mention of it in the code base or in the traditional spot provided by github next to "contributors".

MIT.
You may use it commercially, if that is your issue.

Great, thank you for your prompt response.


Missing output shape for inverse stft

Link
https://www.tensorflow.org/api_docs/python/tf/signal/inverse_stft
Description of issue:
I am missing what the output dimensions are.
Clear description
I do not know what the output dimensions of inverse_stft are. And I do not know how they come to be.
Example Code
audioNp = np.random.random((1,120800)).astype(np.float32)
frame_length = 2048
frame_step = 2048//4
stft = tf.contrib.signal.stft(
    audioNp[0],
    frame_length,
    frame_step)
invstft = tf.contrib.signal.inverse_stft(
    stft,
    frame_length,
    frame_step,
    window_fn = tf.contrib.signal.inverse_stft_window_fn(frame_step))
sess = tf.Session()
stft, invstft = sess.run((stft, invstft))
print(invstft.shape)

The output shape of invstft is then (120320,). And I don't know how it got there. In my opinion it should be again (120800).

I have tried on colab with TF version1.14 , 1.15.0-rc0 and see output shape of invstft is (120320,).Thanks!

With trial and error I found out that the ouput shape seems to be: audioNp[0].shape//frame_step*frame_step
So rounded down to a multiple of frame_step. But can anyone confirm this?

Thanks for the report! This is actually intended behavior. The padding behavior of tf.signal.stft can be set via the pad_end keyword argument. If you set pad_end=True, you'll get a (1, 120832) Tensor out, while the default (pad_end=False), will truncate the signal to only compute the STFT over full frames and give you (1, 120320). There's no way we can give you the exact size of your input back because the input to tf.signal.inverse_stft is an integer number of frames. Hope that helps :).


Remove unnecessary trait bounds from the definition of combinators

note: one potential downside is that this makes it a breaking change to re-add these bounds in the future if the internals of the type were to change in such a way that these bounds were needed.


Add analyzer warnings for classes holding FFI pointers to either release ownership or call the dispose method.

Hi to the team.
Today I was writing a Flutter FFI library and it hit me:
What if there was a way to annotate a class as holding native resources
and this annotation reminds you that you needing to call some special method to manage the pointer inside.
Say, imagine I have a structure of a class like the following
@holdsNativeResources
abstract class  Disposable<Pointer<T>> {
    Pointer<T> ptr;

    Disposable(this.ptr);

    Pointer<T> releasePointer() {
        var copy = ptr;
        ptr = Pointer.nullptr;
        return copy;
    }

    void dispose() {
        <alloc>.free();
        ptr = Pointer.nullptr:
    }

}

And then, say, the analyzer would be able to tell if you haven't either released the pointer (ie. changed ownership of it) or disposed of it properly.
I know this work might conflict with the Dart finalizers, but I think that it would be a first step towards more-correct FFI usage.

@dcharkes @cskau-g

And then, say, the analyzer would be able to tell if you haven't either released the pointer (ie. changed ownership of it) or disposed of it properly.

Dart does not have an ownership tracking type system like Rust does. How do you propose the Dart analyzer would know when to alert you?

It would be great if it worked like Rust borrow checker, but right now I
would like it to work like the don't forget to close sinks lint, maybe?
El jue., 22 de jul. de 2021 02:44, Daco Harkes @.***>
escribió:

And then, say, the analyzer would be able to tell if you haven't either
released the pointer (ie. changed ownership of it) or disposed of it
properly.
Dart does not have an ownership tracking type system like Rust does. How
do you propose the Dart analyzer would know when to alert you?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/dart-lang/sdk/issues/46649#issuecomment-884717391,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABSTWTG6OKITQW5ITBSBXWDTY7D7BANCNFSM5AQWS3XQ
.

You mean a lint using the static scope? Like if you forget to do it before the end of a method body?
How about you use the Arena from package:ffi to do that instead as a coding pattern?
using((Arena arena) {
  // Allocate in arena.
  final pointer = arena<Int64>(2);

  // Use pointer.
});
// Everything in arena is freed.

Yes, I'm already doing that, but the issue is with classes wrapping
pointers to structs and such. I'll get some example code later to
illustrate my point better.
El jue., 22 de jul. de 2021 05:23, Daco Harkes @.***>
escribió:

You mean a lint using the static scope? Like if you forget to do it before
the end of a method body?
How about you use the Arena from package:ffi to do that instead as a
coding pattern?
using((Arena arena) {
// Allocate in arena.
final pointer = arena(2);
// Use pointer.
});// Everything in arena is freed.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/dart-lang/sdk/issues/46649#issuecomment-884806663,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABSTWTABKMRXYT3N3PORQULTY7WR3ANCNFSM5AQWS3XQ
.

  static FrequenciesData loadFromAudioFile(String path) {
    const fftSize = 1024;
    const frameCount = fftSize;
    var arena = Arena(); // To avoid memory leaks
    var runningAverages = Float32List(fftSize);
    try {
      // Instantiates a ma_decoder struct and wraps it on this object.
      var decoder = MiniAudio.openDecoder(path); // I would like that the analyzer marked this if i'm missing a call to closeDecoder() or any other method I would use to free the C struct, or change ownership.
      // Getting the size of the frame in bytes AKA numChannels * sampleSize;
      var frameSize = decoder.frameSize;
      // My FFI returns -1 if it finds an error.
      assert(frameSize > 0, "Size can't be negative!");
      // Array for 256 frames (basically a holder array for the current samples)
      var sampleArray =
          arena.allocate(frameSize * frameCount, alignment: decoder.sampleSize);
      // Checks for the output format from MiniAudio
      var sampleFormat = decoder.decoder.ref.outputFormat;
      // nullable ptr. non-null if we need to do sample conversion (FFT lib uses float32).
      Pointer<Float>? sampleConvPtr;
      if (sampleFormat != ma_format.ma_format_f32) {
        sampleConvPtr = arena.call<Float>(frameCount);
      }
      var framesProcessed = 0; // Used to keep track
      // Used since miniaudio read_frames indicates EOF by returning you less that the requested amount of frames.
      var readFrames = 0;
      var fftFrameCount = 0;
      var sampleRate = decoder.decoder.ref.outputSampleRate;
      do {
        // Write frames to samplesArray.
        readFrames = decoder.readFrames(
            sampleArray.cast<Uint8>(), frameSize * frameCount, frameCount);
        framesProcessed += readFrames; // Add the recorded frames.
        late ArrayComplex fftResult;
        if (sampleConvPtr != null) {
          // If it needed conversion, convert it to f32
          extractF32Samples(sampleFormat, sampleArray.cast<Uint8>(), readFrames,
              sampleConvPtr);
          var samples = Array(sampleConvPtr.asTypedList(readFrames));
          fftResult = rfft(samples, n: fftSize);
        } else {
          // Else set the input to the window function to the raw samples.
          var samples = Array(sampleArray.cast<Float>().asTypedList(readFrames));
          fftResult = rfft(samples, n: fftSize);
        }
        // Transform Real FFT results to dBFS
        var absFft = arrayComplexAbs(fftResult);
        var normalizedFft = arrayDivisionToScalar(absFft, 0.5);
        var dbArray = arrayLog10(normalizedFft);
        var dbfs = arrayMultiplyToScalar(dbArray, 20);
        for(var i = 0; i < dbfs.length; i++){
          var divisor = fftFrameCount + 1;
          var newAverage = (runningAverages[i] * fftFrameCount + dbfs[i]) / divisor;
          if(newAverage <= double.negativeInfinity) {
            runningAverages[i] = -90;
          } else {
            runningAverages[i] = newAverage;
          }
        }
        fftFrameCount++;
      } while (readFrames == frameCount);
      decoder.closeDecoder();
      return FrequenciesData(
          sampleRate: sampleRate,
          data: runningAverages.take(fftSize ~/ 2).mapIndexed((i, e) => FrequencyIntensityPair(
                  getFrequency(i, sampleRate, fftSize), e))
              .toList());
    } catch (e) {
      print('Error received from library: $e');
      rethrow;
    } finally {
      arena.releaseAll();
      print("Should execute.");
    }
  }

This example is a little bit long as I'm actually writing this code for some audio processing in Flutter, but in order of easiest and most reachable would be:

Annotate some of the class methods (that is the output of MiniDecoder.openDecode()) as either of those should be called, kinda the same way sinks do now (so that I mark closeDecoder() and releaseDecoderPtr() methods on that obj, and if I don't call them, then the analyzer warns me).
Have some classes that implement ownership and/or refcount of the raw ptr's inside the class instances (maybe something like this) (please don't take this as something completely thought-out) :

class MiniDecoder extends ffi.Refcounted {
 Pointer<ma_decoder_t> _decoder;
 MiniDecoder(this._decoder);
@override
List<Pointer> ownedPtrs => [_decoder];
}

Would you still care about this lint if Dart providers https://github.com/dart-lang/sdk/issues/45455 a finalizer api suitable for FFI?

To be honest, the proposal went a little bit over my head in how nuanced the situation was. I would probably not care about the lint if Dart provided an easy to use finalizer API, but it's definitively worth checking out.

triaged as P3 not because it isn't potentially important, but because an analyzer change may not be the right way to solve the problem.

I agree. That sounds more like a lint.


Preventive removal of the apache .pid file needed for restarting an existing container

Hi Brian,
thanks a lot for sharing your blog https://www.brianchristner.io/how-to-automate-docker-builds-end-to-end/. It is a very good step by step introduction to the topic.
I was following your instructions using a local Tutum node (following both variants described in https://github.com/tutumcloud/node). It worked well at first, but then I have realized that Tutum has a problem with restarting the containers, once they had been stopped non-gracefully (e.g. because of a reboot of my notebook hosting the Tutum node). This is, because a .pid file is not removed, if the container is stopped non-gracefully and I got the error like "httpd dead but pid file exists", I believe.
In /run.sh, I have added a line to remove the pid file in order to mitigate the problem. After that change, Tutum is able to restart the container with no problem.
Best regards,
Oliver

Thanks for finding the bug and making the change!


feat: remove placeholder option from globalConfig

Issue number: ADDON-71391
Summary
Changes

Please provide a summary of what's being changed

The placeholder option has been removed from all the applicable places. The test files that were having this option, it is removed from them as well. If the placeholder isn't present, it passes and the schema version would be updated to 0.0.8 as we have changed schema.json as well.
User experience

Please describe what the user experience looks like before and after this change

If the developer is using placeholder, their build would execute with the error pointing to resolution, otherwise it would continue as is it was working before.
Checklist
If your change doesn't seem to apply, please leave them unchecked.

[x] I have performed a self-review of this change
[x] Changes have been tested
[x] Changes are documented
[x] PR title follows conventional commit semantics

@vtsvetkov-splunk @soleksy-splunk can one of you please take a look at this PR and make necessary FE changes?

sure, i can take that

removed placeholder options from ui folders


Can't inject InputElement

In angular_forms, we have a number of directives that only match an InputElement. However, because of limitations of the DI system, we can only inject a HtmlElement and cast to InputElement. Otherwise, we get No provider found for InputElement error.
Example:
@Directive(selector: `input[ngModel]`)
class MyInputModel {
  final InputElement _element;

  MyInputModel.fails(this._element); // This fails!!

  MyInputModel.works(HtmlElement element)
      : _element = element as InputElement; // This works!
}

To clarify, this would be a new issue - we would treat any subclass of Element as valid.

To clarify further, we only want to do this for injection. Queries are trickier, because we currently don't store elements as anything but Element or HtmlElement. For example, to query InputElement we would need the (static) type to be of InputElement.


Write and read special matrices, e.g. Diagonal, Symmetric, Transpose

resolve #55

Codecov Report
Attention: Patch coverage is 91.17647% with 3 lines in your changes are missing coverage. Please review.

Project coverage is 93.52%. Comparing base (f73aa94) to head (6ff932f).

Files
Patch %
Lines

src/dataecon/DataEcon.jl
76.92%
3 Missing :warning:

Additional details and impacted files
@@            Coverage Diff             @@
##           master      #66      +/-   ##
==========================================
- Coverage   93.60%   93.52%   -0.08%
==========================================
  Files          24       24
  Lines        4753     4759       +6
==========================================
+ Hits         4449     4451       +2
- Misses        304      308       +4

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


[Enhancement/Feature] [Integration] [argoCD] : allow for Cluster name override

Motivation
We are using argoCD in an embedded mode, that means we have an instance of ArgoCD running in each of our clusters.
each cluster définition comes with name and a server values.

The server value is always the same as it is internal to the cluster but the cluster name changes in order to be able to identify it.
The current integration uses the api/v1/applications api to list all applications and uses the cluster name to link the app to a cluster.
Unfortunately the API does not returns the cluster name, see the issue I created on argoCD side.
Proposed Changes
Would it be possible to allow an override of cluster name in the integration to work around that issue ?
Adding a new config for the user to specify the cluster name.
Are you willing to submit a PR?

[X] I'm willing to submit a PR!

Hey @sgandon 👋🏼 nice meeting you glad to see you are using the ArgoCD integration 🤩
Can you please elaborate on how you installed your ocean ArgoCD integration, as I am less familiar with the embedded mode.
Is it still one installation per all of your clusters or one per cluster due to the embedded behavior?
If it's one per cluster, you can just add the cluster prefix to the mapping configuration of each integration through the UI.
cluster: "prod-cluster"
If it's still one installation for all of the cluster then I am not quite sure how you want to use the override inside the code.

Hello @Tankilevitch,
ArgoCD services are installed in each of our clusters.
For the Port integration I create a single Data source and since I am just doind tests I launched the docker image integration locally but eventually this shall be installed as a pod in our cluster.
For my test I worked around the issue by doing exactly what you suggested and hardcoded the cluster name in the port mapping.
But I don't want to create a new port data source for each of my clusters (too many) and override them all with each cluster name.
I'd rather have a way on the ocean process to override the cluster name. That way I'll have one single data source of argoCD on Port side otherwise I'll have to duplicate all mapping changes I need to be doing for all argocd datasources.

Hey @sgandon .
Thanks for clarifying 🙏
Eventually ocean is running jq to process the entities, so if you want to keep the same mapping for all datasources you can configure an environment variable for each of the installations and use it inside the JQ.
resources:
  - kind: project
    selector:
      query: 'true'
    port:
      entity:
        mappings:
          identifier: .path_with_namespace | gsub(" "; "")
          title: .name
          blueprint: '"service"'
          properties:
            url: .web_url
            readme: file://README.md
            description: .description
            language: .__languages | to_entries | max_by(.value) | .key
            cluster: env | .CLUSTER_NAME

for more information see this issue thread - https://github.com/port-labs/ocean/issues/556#issuecomment-2061275926
I have also created a canny feature request about your use case, so feel free to upvote and leave a comment.

thanks a lot @Tankilevitch for your response, the environment variable is an awesome trick and I really suggest you add this  to some port documentation !


autogenerate creates alter_column instructions every time

After autogenerating and applying the generated migration with the upgrade command, a second autogeneration run yields an alter_column for every column using an ENUM:
op.alter_column('buffer_package', 'download_status',
               existing_type=postgresql.ENUM('NOT_CREATED', 'CREATED', 'DOWNLOADED', 'CONFIRMED', 'CREATED_FOR_DOWNLOAD', 'CREATED_FOR_UPLOAD', 'UPLOADED', name='bufferpackagestatus'),
               type_=sa.Enum('NOT_CREATED', 'CREATED', 'DOWNLOADED', 'CONFIRMED', 'CREATED_FOR_DOWNLOAD', 'CREATED_FOR_UPLOAD', 'UPLOADED', name='bufferpackagestatus', schema='con_mon'),
               existing_nullable=False,
               schema='con_mon',
               postgresql_using='download_status::bufferpackagestatus')

how can I fix this or work around it?
It seems that it always changes the type from "postgresql.ENUM" to "sa.Enum" but the Type is either not applied or not red correctly

the Log shows:
INFO  [alembic.autogenerate.compare] Detected type change from ENUM('NOT_CREATED', 'CREATED', 'DOWNLOADED', 'CONFIRMED', 'CREATED_FOR_DOWNLOAD', 'CREATED_FOR_UPLOAD', 'UPLOADED', name='bufferpackagestatus') to Enum('NOT_CREATED', 'CREATED', 'DOWNLOADED', 'CONFIRMED', 'CREATED_FOR_DOWNLOAD', 'CREATED_FOR_UPLOAD', 'UPLOADED', name='bufferpackagestatus', schema='con_mon') on 'buffer_package.download_status'

is it a schema problem?

Please provide first migration for more context

As I can see there is something wrong with schema

sa.Enum('NOT_CREATED', 'CREATED', 'DOWNLOADED', 'CONFIRMED', 'CREATED_FOR_DOWNLOAD', 'CREATED_FOR_UPLOAD', 'UPLOADED', name='bufferpackagestatus', schema='con_mon').create(op.get_bind())
op.create_table('buffer_package',
    sa.Column('customer_id', sa.UUID(), nullable=False),
    sa.Column('file_name', sa.String(), nullable=False),
    sa.Column('checksum', sa.String(), nullable=False),
    sa.Column('download_status', postgresql.ENUM('NOT_CREATED', 'CREATED', 'DOWNLOADED', 'CONFIRMED', 'CREATED_FOR_DOWNLOAD', 'CREATED_FOR_UPLOAD', 'UPLOADED', name='bufferpackagestatus', schema='con_mon', create_type=False), nullable=False),
    sa.Column('additional_properties', sa.JSON(), server_default='{}', nullable=False),
    sa.Column('created_at', sa.DateTime(), server_default=sa.text('now()'), nullable=False),
    sa.Column('id', sa.UUID(), server_default=sa.text('gen_random_uuid()'), nullable=False),
    sa.ForeignKeyConstraint(['customer_id'], ['con_mon.customer.id'], ),
    sa.PrimaryKeyConstraint('id'),
    schema='con_mon'
    )
op.create_index(op.f('ix_con_mon_buffer_package_id'), 'buffer_package', ['id'], unique=False, schema='con_mon')

in my env file I have:
def run_migrations_online() -> None:
    """Run migrations in 'online' mode.

    In this scenario we need to create an Engine
    and associate a connection with the context.

    """

    def process_revision_directives(context, revision, directives):
        if config.cmd_opts.autogenerate:
            script = directives[0]
            if script.upgrade_ops.is_empty():
                directives[:] = []
                print('No changes in schema detected.')

    connectable = engine_from_config(
        config.get_section(config.config_ini_section, {}),
        prefix="sqlalchemy.",
        poolclass=pool.NullPool,
    )
    print(f'\n{target_metadata.schema=}\n')
    with connectable.connect() as connection:
        context.configure(
            connection=connection,
            target_metadata=target_metadata,
            version_table_schema=target_metadata.schema,
            include_schemas=True,
            include_name=include_name,
            process_revision_directives=process_revision_directives
        )
        connection.execute(text(f"CREATE SCHEMA IF NOT EXISTS {target_metadata.schema}"))
        connection.execute(text(f"SET search_path TO {target_metadata.schema}"))

        with context.begin_transaction():
            context.run_migrations()
            ```

Seems like the root of the problem is with search_path. If that is true it is outside of our responsibility

I will dig into that and keep you posted. Thank you for the pointer!

removing the search_path line solved this even but i am not sure why, because i strongly believe i put it in on purpose to make it work.
But anyway, thank you for the quick replies and for this Tool!
def run_migrations_online() -> None:
    """Run migrations in 'online' mode.

    In this scenario we need to create an Engine
    and associate a connection with the context.

    """

    def process_revision_directives(context, revision, directives):
        if config.cmd_opts.autogenerate:
            script = directives[0]
            if script.upgrade_ops.is_empty():
                directives[:] = []
                print('No changes in schema detected.')

    connectable = engine_from_config(
        config.get_section(config.config_ini_section, {}),
        prefix="sqlalchemy.",
        poolclass=pool.NullPool,
    )
    print(f'\n{target_metadata.schema=}\n')
    with connectable.connect() as connection:
        connection.execute(text(f"CREATE SCHEMA IF NOT EXISTS {target_metadata.schema}"))
        context.configure(
            connection=connection,
            target_metadata=target_metadata,
            version_table_schema=target_metadata.schema,
            include_schemas=True,
            include_name=include_name,
            process_revision_directives=process_revision_directives
        )
        # connection.execute(text(f"SET search_path TO {target_metadata.schema}"))

        with context.begin_transaction():
            context.run_migrations()


[Docs]: Getting started page is missing instructions for installing @serwist/precaching and @serwist/sw

Provide environment information
I visited https://serwist.pages.dev/docs/next/getting-started
Describe the problem
At Step 2 - creating sw.ts - VSCode noticed that there are imports for @serwist/precaching and @serwist/sw but I hadn't yet installed those (Step 1 missed them)
Expected behavior
Step 1 should be complete
Screenshots (if relevant)
No response
Additional information (if relevant)
Once I installed those two packages my errors went away

Fixed :) Thanks for reporting!


Perf Analysis UI Graph bug

Looks like the y-axis label for perf graph is getting overlapped with values.

I am not sure how to fix this. Unless I change the units to seconds?


Add LocalRF

Meta unveiled LocalRF code this week: https://github.com/facebookresearch/localrf
The main features are two:

it handles with big scenes
it can predict scene without known camera positions

How possible to implement this one in nerfstudio?

Duplicate #2099


Fix a (possible) bug in surface BC

Looking at the animation of qt evolution, I think there is something wrong there

I think we just have q_surf=0.016 when it should be q_surf=0.015. Otherwise we can try a BC of "extrapolate" at the bottom too.

Fixed in commit d5af19277521f1714cef859d32b5a9f6a15f7fec


The Documentation Website returns a 404 error

https://neurotechx.github.io/eeg-expy/index.html
📝 Provide a description of requested docs changes
Configure github pages to correctly pull from the gh_pages branch.
What is the purpose and what should be changed?
To allow the documentation link provided in the README.md file to correctly connect to the documentation website.

See PR #275


Increase platform detection

Hey,
I am not good with rust but I noticed that you had the platform detection JSON hardcoded in platform.rs.
It would be nice to provide a separate JSON file that will contain all the information. This way people can also use their custom list.
If you want you can take a look at one of my tools i.e takeover in that I have a file named providers.json which was made by scraping the can-i-take-over-xyz repository's readme combining it with subjack's fingerprint.json and subOver's provider.json

@mzfr Thanks a lot for sharing it here, I am going through all the URLs that you've provided and will update the remote repository (https://github.com/Pentester-Nepal/Subdomain-Takeover-Signatures/)
AND since, I'd simply copy pasted the JSON content within platform.rs and according to @TheBinitGhimire's README.md it was taken from (https://github.com/EdOverflow/can-i-take-over-xyz) that's why, I've just credited a single source. As soon as I include it from all of the destinations you've provided. I'll credit them too (including your repo. ngl, loved it!) 😃

Hello @mzfr,
As @naryal2580 mentioned earlier, the issue you have opened appears to be resolved now, since users can now remotely fetch the latest signatures from the remote repository, and also add their own custom signatures in the signatures.json file to use with NtHiM.
If you are wondering where you can find the signatures.json file for NtHiM, then the following information shall be helpful for you:

Microsoft Windows: %LOCALAPPDATA%\NtHiM\signatures.json
Linux/*BSD: ~/.cache/NtHiM/signatures.json
macOS: ~/Library/Caches/signatures.json

With these information, I believe your issue has been resolved now, and I will be closing it for now.
Thank You opening up this issue, and providing an opportunity to include useful functionalities within NtHiM!
Thanks,
@TheBinitGhimire

Hello @mzfr,
As @naryal2580 mentioned earlier, the issue you have opened appears to be resolved now, since users can now remotely fetch the latest signatures from the remote repository, and also add their own custom signatures in the signatures.json file to use with NtHiM.
If you are wondering where you can find the signatures.json file for NtHiM, then the following information shall be helpful for you:

Microsoft Windows: %LOCALAPPDATA%\NtHiM\signatures.json
Linux/*BSD: ~/.cache/NtHiM/signatures.json
macOS: ~/Library/Caches/signatures.json

With these information, I believe your issue has been resolved now, and I will be closing it for now.
Thank You for opening up this issue, and providing an opportunity to include useful functionalities within NtHiM!
Thanks,
@TheBinitGhimire

Hello @mzfr,
As @naryal2580 mentioned earlier, the issue you have opened appears to be resolved now, since users can now remotely fetch the latest signatures from the remote repository, and also add their own custom signatures in the signatures.json file to use with NtHiM.
If you are wondering where you can find the signatures.json file for NtHiM, then the following information shall be helpful for you:

Microsoft Windows: %LOCALAPPDATA%\NtHiM\signatures.json
Linux/*BSD: ~/.cache/NtHiM/signatures.json
macOS: ~/Library/Caches/signatures.json

With these information, I believe your issue has been resolved now, and I have closed it for now.
Thank You opening up this issue, and providing an opportunity to include useful functionalities within NtHiM!
Thanks,
@TheBinitGhimire


Added Italian translation

Added the italian translation

Thank you! This will be in the next release.

Now live: https://onetimesecret.com/?locale=it_IT
Let me know what name/link you would like on the translations page (or submit a PR):
https://onetimesecret.com/translations

You can use Giuseppe Pignataro (fastbyte01)
Il 24 Nov 2017 02:13, "Delano Mandelbaum"<EMAIL_ADDRESS>ha
scritto:

Now live: https://onetimesecret.com/?locale=it_IT
Let me know what name/link you would like on the translations page (or
submit a PR):
https://onetimesecret.com/translations
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/onetimesecret/onetimesecret/pull/72#issuecomment-346716787,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AQFoqrhW8AydeZoMJbBqx6JQDQ3NbRTkks5s5hg1gaJpZM4PsUhz
.

Done and done. Thanks for helping make One-Time secret better.


Consolidate transform_fn and collate_fn in torch.DataLoader

These two functions are basically the same
https://github.com/eto-ai/rikai/blob/master/python/rikai/torch/data.py#L41-L42
torch.data.DataLoader uses collate_fn https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader

I just looked again, the transform_fn is applied to a single source record inside the parallel prefetch thread (e.g., for sampling frames from a video) while the collate_fn will be applied to a batch of output records out of the queue. Maybe we can change the name and document it better? Do you have a better name suggestion?

I just looked again, the transform_fn is applied to a single source record inside the parallel prefetch thread (e.g., for sampling frames from a video) while the collate_fn will be applied to a batch of output records out of the queue. Maybe we can change the name and document it better? Do you have a better name suggestion?

i've seen transform somewhere in pytroch doc.
https://pytorch.org/tutorials/beginner/data_loading_tutorial.html#iterating-through-the-dataset
Can we make them compatible?

i've seen transform somewhere in pytroch doc.
https://pytorch.org/tutorials/beginner/data_loading_tutorial.html#iterating-through-the-dataset
Can we make them compatible?

#62 will kick the can down the road for now

#62 will kick the can down the road for now


Enhancement: Reduce time for -> Sleeping 5 minutes to prevent getting blocked

It'll be nice to reduce waiting time to 1 or 2 mins, if you have a lot of people to unfollow this gets really annoying :/

It'll be nice to reduce waiting time to 1 or 2 mins, if you have a lot of people to unfollow this gets really annoying :/

This time is based in my expierence, you could tweak it a lower it. But you will get temp ban ans you could not scan more users in maybe 10-15 minutes. So I think its not worth. But if you want to do it, there is a constant to change the sleep time.

@davidarroyo1234 How about to change number of people unfollowed to 20-30 and then wait 5min? Will that work?

@davidarroyo1234 How about to change number of people unfollowed to 20-30 and then wait 5min? Will that work?

I ran this measurement based in my on account,but you could try to unfollow 20-30 manually in your account and see if you could increase this. Be aware that the ban its silent, so it will show as unfollowed but if you refresh the page it will show that you still follow him.

hey i tried many ways to alter this like creating a global "wake" flag, but i kept breaking code, ig detects that and softbans that quicker than anything. instead of sleeping for 5 minutes, any other way to have me give an "input" or prompt for it to "wake" and run as opposed to a timed interval of 3e5? i think this is less robotic. my global flag method did not work.

hey i tried many ways to alter this like creating a global "wake" flag, but i kept breaking code, ig detects that and softbans that quicker than anything. instead of sleeping for 5 minutes, any other way to have me give an "input" or prompt for it to "wake" and run as opposed to a timed interval of 3e5? i think this is less robotic. my global flag method did not work.

You can change this millisecond to what ever you prefer. But keep in mind that if you lower the value too much you will get softban and you will not notice it.

Anybody can help me? Where I can change the delay script? I copy the script to console, I cannot see where is loading this script. Sorry I'm not expert in React and JavaScript.

the delay script? I copy the script to console, I cannot see where is loading this script. Sorry I'm not expert in React and JavaScript.

@davidarroyo1234 How about to change number of people unfollowed to 20-30 and then wait 5min? Will that work?

It'll be nice to reduce waiting time to 1 or 2 mins, if you have a lot of people to unfollow this gets really annoying :/

I have published a new update, now you can configure it clicking in the setting button. Use it at your on risk. This can lead in a ban


Event: appland.appmap/plugin/remote_recording:stop

:tada: This issue has been resolved in version 0.18.0 :tada:
The release is available on:

v0.18.0
GitHub release

Your semantic-release bot :package::rocket:


Updated pre-commit hook

clang-format check may actually fail for two reasons:
1. There are no modified files, and
2. There is nothing to do

The old version of the pre-commit only checked for 1. However, if
clang-format check passed, it failed.
My bad.

Current coverage is 95.69% (diff: 100%)

Merging #175 into master will not change coverage

@@             master       #175   diff @@
==========================================
  Files           375        375
  Lines         27983      27983
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
  Hits          26778      26778
  Misses         1205       1205
  Partials          0          0

Powered by Codecov. Last update f36e33c...759b827


Python 3.13 support need

Python 3.13 support need

Hi @patrizok  -- Please see the tracking for Python 3.13 support in our JIRA system:  PYCBC-1613
We will address Python 3.13 support more quickly than we did Python 3.12, but some changes in Python's C-API mean we need to tweak our C/C++ bindings and that causes a bit of delay for adding new wheels (small team, sprint planning, etc. 😅).  I have made a note to circle back around and update this issue once we have added 3.13 support.

Hi @patrizok  -- PYCBC-1613 has been resolved and v4.3.4 has been released which includes Python 3.13 support.


Documentation, and startup script adjustment for logs plus env changes

Hello
Main changes are documentation with compose examples
Additionally some changes main changes
Added the variable LOG_RETENTION which allows users to set in days how long log files will be kept for.
This is NOT a breaking change as if this variable is not specified it will default to 30 days.  This change changes the launch script and env reference files.
This default can be configured and changed inside the .sh script
I have also pulled two variables out from the stack env file and placed them into the compose file so they can be set as a per server setting. This allows one stack to have multiple servers that can have independent config files set per service, and managing from a single compose file.  It can even mix and match modded and unmodded.
The two variables are
environment:
CONFIG_FILE: Config.json
#ADDITIONAL_ARGUMENTS:
I will add a multi service stack example in the future.

aweosme


chore(deps): update packages

Types of changes

[x] Update npm packages
[ ] Bug fix (non-breaking change which fixes an issue).
[ ] New feature (non-breaking change which adds functionality).
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected).
[ ] Performance improvement
[ ] Refactoring (so no functional change)
[ ] Other (docs, build config, etc)

Checklist:

[x] I've added tests for my code. (most of the time mandatory)
[ ] I have added an entry to the changelog. (mandatory)
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 100.00%. Comparing base (23eaad8) to head (952016a).

Additional details and impacted files
@@            Coverage Diff            @@
##              main      #110   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files          152       152
  Lines         7476      7476
  Branches       688       688
=========================================
  Hits          7476      7476

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Disable editing for ID field

Once a record has been created, it is not possible alter the ID in Solr.  However, GEOMG allows a user to edit this field.
We should make the field uneditable after the record is created.

Reopening to bring this behavior back.


Multiplayer Networking

Hello, how is multiplayer networking for weber within unity handled? Can you use photon or is there another more simple way of implementing it? Also, I was messing around with Mozilla hubs and it seems like that would be useful it it were incorporated into Unity WebVr build. Looking for any input on a road to go down on this. Thanks!

You could use websocket sharp and Node.js.

Focusing on just the basics for now, but would love to incorporate web-based multiplayer networking. Unsure if existing Unity solutions will work within the browser as-is. Would love to hear if anyone has any luck with this.


turkish locale added

turkish locale added.

Thank you!

sorry for my ruby ignorance.
i will add needed date time specs too.
On Sep 22, 2015 2:19 PM, "Tomasz Gieniusz"<EMAIL_ADDRESS>wrote:

Thank you!
—
Reply to this email directly or view it on GitHub
https://github.com/tomgi/git_stats/pull/60#issuecomment-142257000.


API Registry search

Hey people of Readme.io :)
I was wondering if there an endpoint or some development that can be done to find all the possible registries out there ie: "opensea/v2.0#3pfh7pm2nkakbd".
It be good to know all the possible installation out there.
Anyhow great job!
I will be more than happy to build such iteration.

Thanks :)


add option to validate / not validate the model

Is your feature request related to a problem? Please describe.
Even after several retries there are cases when the LLM is not able to return data that the validators will validate. For those case it would be nice to have a backdoor where the .create() does not return an error. Instead I would like to be returned what the LLM returned, even if the data has not been validated.
Describe the solution you'd like
An option when calling .create() to choose whether the data is validated or not.
Describe alternatives you've considered

Option where in case of error the values of each field are set to their default value defined in the Pydantic model.

Additional context

Instead I would like to be returned what the LLM returned, even if the data has not been validated.

https://python.useinstructor.com/#returning-the-original-completion-create_with_completion
user, completion = client.chat.completions.create_with_completion(
    model="gpt-4-turbo-preview",
    messages=[
        {"role": "user", "content": "Create a user"},
    ],
    response_model=User,
)

user  contains the Object validated (or not)
completion contains the raw response from LLM.
If you don't use max_retries, then only one validation will be attempted.


Input is too long for requested model

I tried many times, got the following error
{
"errorType": "ValidationException",
"errorMessage": "Input is too long for requested model.",
"name": "ValidationException",
"$fault": "client",
"$metadata": {
"httpStatusCode": 400,
"requestId": "16fb0962-139d-4c70-ba9c-602d9630e854",
"attempts": 1,
"totalRetryDelay": 0
},
"stack": [
"ValidationException: Input is too long for requested model.",
"    at de_ValidationExceptionRes (/var/task/index.js:84:74209)",
"    at de_CommandError (/var/task/index.js:84:71685)",
"    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)",
"    at async /var/task/index.js:6:161455",
"    at async /var/task/index.js:15:1230",
"    at async /var/task/index.js:14:15753",
"    at async /var/task/index.js:65:116368",
"    at async A_.callLLM (/var/task/index.js:162:1082)",
"    at async A_.getTopics (/var/task/index.js:162:288)",
"    at async T_.processEvent (/var/task/index.js:162:1988)"
]
}

Hello @gutomo ,
Thank you for flagging this issue.
Would it be possible for you to share the document that you are testing with? Or alternatively, could you share the size of the document that you are using to generate the assessment?

Hello @mriccia
I tried using this document (https://d1.awsstatic.com/whitepapers/wordpress-best-practices-on-aws.pdf) with model Claude haiku or sonnet, but both attempts failed with the error message 'Input is too long for requested model.' I used the AWS T3 small instance for OSS.
By the way, I also tried a similar app (https://github.com/aws-samples/serverless-exam-generator-from-your-lecture-amazon-bedrock),
with the same document  and it was successful.


Choosing a sign-in model for microsoft 365 url hyperlink returns http 404

The Choosing a sign-in model for Microsoft 365.
URL returns a 404 - not found

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 0df486c9-1fa9-bf49-d702-372b19d20639
Version Independent ID: 0df486c9-1fa9-bf49-d702-372b19d20639
Content: Authenticating to Power Platform services - Power Platform
Content Source: power-platform/admin/security/authenticate-services.md
Service: power-platform
Sub-service: admin
GitHub Login: @lancedMicrosoft
Microsoft Alias: lanced

Link points to
https://www.microsoft.com/microsoft-365/blog/2014/05/13/choosing-a-sign-in-model-for-office-365/
Link at bottom of page points to same topic, but has /en-us in it.
https://www.microsoft.com/en-us/microsoft-365/blog/2014/05/13/choosing-a-sign-in-model-for-office-365/
Once I click on the longer link, the link without /en-us in it seems to work.
But we should try to find resource that is newer than 2014.

Thanks for your comment, @KeysAU. This issue has been assigned to the doc author for review.


Enhance Authentication Error Handling in nzgo Driver

Problem : Inconsistent Authentication Error Handling
Currently, there is a scenario where authentication errors are being handled within the 'connNextResultSet' function's notice message case 'N'. This situation occurs when an authentication error arises during the connection process.
Solution :
To create a more consistent and accurate error handling process, I have made specific changes. Authentication errors that occur during the connection process are now directly managed within the "Conn_authenticate' function. This enhancement ensures a more reliable error handling mechanism throughout the driver.
Testing :
I thoroughly tested the driver by connecting to a Netezza database and intentionally causing authentication errors. With the changes in place, the driver now effectively handles the specific error during authentication.

@RohitChelluru06 Remove extra spaces

@RohitChelluru06 Remove extra spaces

Fix looks good to me. Approved.


ECCOtour description on GitHub references old Reemergence.jl info

In repository overview page, GitHub has outdated information describing ECCOtour.jl. Where can this be edited?

Editable in the About tab on the main page


fix: break all words in typography

:tada: This PR is included in version 5.7.1 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Upgrade to PHP 7.4

Closes #140

cc @JanMikes I'm upgrading page to PHP 7.4
Could you update the image version as well? It's the only one failing now

Please next time do not merge it. If there is something failing it is failing for reason and has to be fixed first.
Merging failing PR can cause a lot of troubles.

Ok.
I don't use Docker and I didn't find any version in dockerfile.


[Mac app] open file from command line

Such as:
mdp test.md

:+1:  on this request  also Tabs more that one file open at a  time?

I am busy with version 2.0 which supports multiple tabs. It will be released this month.

Multiple tabs feature has been implemented.
In order to launch the app from command line, you can add an alias:
alias mdp="open -a /Applications/Markdown\ Plus.app"

Then you can:
mdp README.md

or just
mdp


Need a way to specify Snappy install path

There doesn't seem to be a way to tell the build system that Snappy is
installed in a non-standard location (e.g. in /opt/local using MacPorts).

Original issue reported on code.google.com by electrum on 1 Aug 2011 at 12:06

I used this way, hope this can help
CXXFLAGS="-I ../snappy-1.0.5 -L ../snappy-1.0.5/.libs/" make

Original comment by<EMAIL_ADDRESS>on 10 Nov 2012 at 2:11


IncludeTotalCount on UsePaging attempts to reuse disposed IClientSessionHandle when we add UseServiceScope to the query

Product
Hot Chocolate
Version
13.5.1
Link to minimal reproduction
https://github.com/luiscarreira/HCScopeError
Steps to reproduce
Using MongoDb as the datastore and expect to use the IClientSessionHandle from Mongo to enable the usage of Transactions.
public class BaseRepository<T> : IBaseRepository<T> where T : BaseEntity
{
    private readonly IMongoCollection<T> collection;

    private readonly IClientSessionHandle _sessionHandle;

    public BaseRepository(IClientSessionHandle sessionHandle, ICatalogContext catalogContext)
    {
        _sessionHandle = sessionHandle;

        if (catalogContext == null)
        {
            throw new ArgumentNullException(nameof(catalogContext));
        }

        this.collection = catalogContext.GetCollection<T>(typeof(T).Name);
    }

    public IExecutable<T> GetAllAsync()
    {
        return this.collection.Find(_sessionHandle, _ => true, default).AsExecutable();
    }
}

Put together the UseServiceScope with UsePaging and IncludeTotalCount = true on the query.
[ExtendObjectType("Query")]
public class ProductQuery
{
    [UseServiceScope(Order = 0)]
    [UsePaging(IncludeTotalCount = true, Order = 1)]
    public IExecutable<Product> GetProductsAsync([Service] IProductRepository productRepository) =>
        productRepository.GetAllAsync();
}

Execute the query
query ProductsAsync {
    productsAsync {
        totalCount
        edges {
            cursor
            node {
                id
                categoryId
                name
                description
                price
                quantity
                category {
                    description
                    id
                }
            }
        }
    }
}

What is expected?
Expected the query to run without issues and ensured that all related database operations were executed in the same MongoDb Session for the entity Product.
What is actually happening?
Product query is executed in the ServiceScope.
The database query (IExecutable) is saved on the HC side and when the total count is requested HC reuses the original Products query to fetch the count, but this happens outside the ServiceScope so the IClientSessionHandler is already disposed and an exception is thrown.
Relevant log output
{
    "errors": [
        {
            "message": "Unexpected Execution Error",
            "locations": [
                {
                    "line": 3,
                    "column": 9
                }
            ],
            "path": [
                "productsAsync",
                "totalCount"
            ],
            "extensions": {
                "message": "Cannot access a disposed object.\nObject name: 'MongoDB.Driver.Core.Bindings.CoreSessionHandle'.",
                "stackTrace": "   at MongoDB.Driver.Core.Bindings.WrappingCoreSession.ThrowIfDisposed()\n   at MongoDB.Driver.Core.Bindings.WrappingCoreSession.get_IsInTransaction()\n   at MongoDB.Driver.MongoCollectionImpl`1.ExecuteReadOperationAsync[TResult](IClientSessionHandle session, IReadOperation`1 operation, CancellationToken cancellationToken)\n   at MongoDB.Driver.MongoCollectionImpl`1.CountDocumentsAsync(IClientSessionHandle session, FilterDefinition`1 filter, CountOptions options, CancellationToken cancellationToken)\n   at MongoDB.Driver.FindFluent`2.CountDocumentsAsync(CancellationToken cancellationToken)\n   at HotChocolate.Data.MongoDb.Paging.FindFluentPagingContainer`1.CountAsync(CancellationToken cancellationToken)\n   at HotChocolate.Data.MongoDb.Paging.MongoCursorPagination`1.CountAsync(IMongoPagingContainer`1 query, CancellationToken cancellationToken)\n   at HotChocolate.Types.Pagination.ConnectionType.GetTotalCountAsync(IResolverContext context)\n   at HotChocolate.Types.Helpers.FieldMiddlewareCompiler.<>c__DisplayClass9_0.<<CreateResolverMiddleware>b__0>d.MoveNext()\n--- End of stack trace from previous location ---\n   at HotChocolate.Execution.Processing.Tasks.ResolverTask.ExecuteResolverPipelineAsync(CancellationToken cancellationToken)\n   at HotChocolate.Execution.Processing.Tasks.ResolverTask.TryExecuteAsync(CancellationToken cancellationToken)"
            }
        }
    ],
    "data": {
        "productsAsync": null
    }
}

Additional context
On the reproduction repository provided, there is an extra custom Field Middleware to create, commit, or abort a MongoDb transaction but this doesn't interfere with the steps. We end up with a disposed IClientSessionHandle anyway.

Hi there, did someone had the opportunity to look into this topic?
I had a look at the source code for the UsePaging with IncludeTotalCount but was unable to locate the point where the query for the TotalCount is called when using the UseServiceScope to try to fix it.


separate sync_container_context to fix problems with the same contextvar token for every context

separate sync_container_context to fix problems with the same contextvar token for every context

@alexanderlazarev0 Hi) there is some problem with new container context.
The problem is that contextvar token variable is the same for all requests.
To fix this I had to separate sync container context

@lesnik512
Do you have a minimal example to reproduce?

@alexanderlazarev0 not right now, in the evening I will make an example

@alexanderlazarev0 Error was the following:
RuntimeError
<Token used var=<ContextVar name='CONTAINER_CONTEXT' at 0x7fade72ab8d0> at 0x7fade06e8c80> has already been used once
Trying to build small reproducible example now

@lesnik512 Yes I was trying to reproduce this, but wasn't able to for now. Is this with fastapi?

@alexanderlazarev0 it's in kafka consumer.
There is a method which called on each message from topic:
    @container_context(build_di_context("write"))
    async def process_one_message(
        self,
        message: timers_schemas.DialogUnreadMessagesTimerEvent,
        _: dict[str, typing.Any] | None = None,
    ) -> None:
         ...

Theres is also another version of this error:
ValueError
<Token var=<ContextVar name='CONTAINER_CONTEXT' at 0x7f3098d479c0> at 0x7f30807b7bc0> was created in a different Context

@lesnik512 Okay seems the exit method of the containter_context is being called twice by different actors.

@alexanderlazarev0 I think, the problem is occured when decorator used: it's initialized only once and _context_token is shared between different contexts

Ah! I got it I think, its a 2 liner:
# container_context class
 def __call__(self, func: typing.Callable[P, T]) -> typing.Callable[P, T]:
        if inspect.iscoroutinefunction(func):

            @wraps(func)
            async def _async_inner(*args: P.args, **kwargs: P.kwargs) -> T:
                async with self: # <- HERE
                    return await func(*args, **kwargs)  # type: ignore[no-any-return]

            return typing.cast(typing.Callable[P, T], _async_inner)

        @wraps(func)
        def _sync_inner(*args: P.args, **kwargs: P.kwargs) -> T:
            with self: # <-- HERE
                return func(*args, **kwargs)

        return _sync_inner

@alexanderlazarev0 seems, like this could work.
But _initial_context must be passed, I think in __call__:
container_context(self._initial_context)

@alexanderlazarev0 But still I have no idea how to reproduce this in tests)

@lesnik512 I think you might be able to reproduce it by calling an @container_context decorated function twice in the same call stack (i need to try just my first guess)

@alexanderlazarev0 seems, like this could work.
But _initial_context must be passed, I think, in __call__:
container_context(self._initial_context)

Yes you are right this will need to be added.
I will let you know if i can reproduce the issue.

@lesnik512 Yes got It and the suggest fix works!
Error example:
from that_depends import providers
from that_depends.container import BaseContainer
from that_depends.injection import inject
from that_depends.providers.context_resources import container_context

def context_resource_creator():
    yield "interesting"

class MyContainer(BaseContainer):
    context_resource = providers.ContextResource(context_resource_creator)
@container_context()
@inject
def some_injected(depth, val = MyContainer.context_resource):
    if depth ==2:
        return val
    return some_injected(depth+1)

if __name__ == "__main__":
    some_injected(1)

@lesnik512 So the question is really then just one of taste:

Keep the current implementation, but now you need to actively annotate correctly with either sync_container_context or container_context
Revert to the old one which is less verbose and arguably easier to use for annotations (less thought required), however it is perhaps not as intuitive to use outside annotationswith container_context() and async with container_context().
Theoretically, both solutions can coexist, but I think 1 clear way to do things is preferred.

@alexanderlazarev0 Nicely done! So I reverted to the previous version, applied fix and wrote some tests https://github.com/modern-python/that-depends/pull/94. Please, take a look


Updated profile and picture for Laurent Bugnion

Docs Build status updates of commit c15639b:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

advocates/laurent-bugnion.yml
:white_check_mark:Succeeded
View

advocates/media/profiles/laurent-bugnion.png
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel