It depends on your organisation. Is the factory the same company (legal entities)? If not you create the factory as a vendor and purchase from the factory. (If you don't want to use Odoo for both companies).

In either case you can customize to have two warehouses and use internal transfer between.

If you are in the same company and you don't want to use manufacturing, the question will be: How will the products come into stock?  This strongly related to accounting and the cost of the products. 

Hi,

For the external purchases that you made from your vendor, you can use the purchase application to record this in the system.

For the items produced in the factory, if you need to handle without using the manufacturing application, you can record the inventory movements in the inventory app by creating a picking. Suppose if you are going to produce some products using some other existing products in the inventory,  you can create an picking type(PT) with name Production and set the destination location as a location of type production. And when create the picking, choose this PT and select internal to production location. This will handle the case of row materials and you can create one more PT to record the production details, which will add the stock to inventory.

Finally using internal transfer you can move the stock across the warehouse.

Thanks